Resumable index build loses side-write multikey state

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 6.0.0, 7.0.0, 8.0.0
    • Component/s: Index Maintenance
    • None
    • Environment:
      Replica set members undergoing clean restarts (e.g. rolling maintenance,
      package upgrades) while hybrid index builds are in progress on
      collections receiving concurrent writes that introduce arrays into
      indexed fields.
    • Storage Execution
    • ALL
    • Hide
      1. Create a collection containing only scalar values for a, for example
        {{ {a: 1}

        }} and {a: 2, on a single-node replica set.

      2. Start createIndexes for {{ {a: 1}

        }} and let the build proceed past
        the collection scan; for a deterministic reproduction enable the
        hangAfterIndexBuildFirstDrain failpoint.

      3. While the build is hung after the first drain, insert the document
        {{ {a: [3, 4]}

        }}. The write is captured only by the side writes
        interceptor.

      4. Cleanly shut down the node. The resumable state is written to disk (log id 4841502) with isMultikey: false.
      5. Restart the node. The build resumes (log id 4841700), drains the
        remaining side writes, and commits.

      Expected result

      The committed index is flagged multikey;
      db.coll.find({a: {$gte: 4, $lte: 3).hint({a: 1})}} returns the array
      document; validate passes.

      Actual result

      explain shows isMultiKey: false on the committed index; the query
      above returns no documents even though {a: [3, 4]} matches;
      {{db.coll.validate(

      {full: true}

      )}} reports the collection as invalid.

      A deterministic end-to-end reproduction is provided by the new jstest
      jstests/noPassthrough/index_builds/resumable_index_build_multikey_from_side_writes.js, and a unit-level regression test is provided by
      MultiIndexBlockTest.ResumeStateIncludesMultikeyFromSideWrites in
      src/mongo/db/index_builds/multi_index_block_test.cpp.

      Show
      Create a collection containing only scalar values for a , for example {{ {a: 1} }} and { a: 2 , on a single-node replica set. Start createIndexes for {{ {a: 1} }} and let the build proceed past the collection scan; for a deterministic reproduction enable the hangAfterIndexBuildFirstDrain failpoint. While the build is hung after the first drain, insert the document {{ {a: [3, 4]} }}. The write is captured only by the side writes interceptor. Cleanly shut down the node. The resumable state is written to disk (log id 4841502) with isMultikey: false . Restart the node. The build resumes (log id 4841700), drains the remaining side writes, and commits. Expected result The committed index is flagged multikey; db.coll.find({a: {$gte: 4, $lte: 3 ).hint({a: 1})}} returns the array document; validate passes. Actual result explain shows isMultiKey: false on the committed index; the query above returns no documents even though { a: [3, 4] } matches; {{db.coll.validate( {full: true} )}} reports the collection as invalid. A deterministic end-to-end reproduction is provided by the new jstest jstests/noPassthrough/index_builds/resumable_index_build_multikey_from_side_writes.js , and a unit-level regression test is provided by MultiIndexBlockTest.ResumeStateIncludesMultikeyFromSideWrites in src/mongo/db/index_builds/multi_index_block_test.cpp .
    • Storage Execution 2026-08-03
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      A hybrid index build indexes documents through two channels: the
      collection scan feeds the bulk builder, and concurrent client writes are
      intercepted and buffered in the side writes table, which is drained into
      the index before commit.

      When a concurrent write makes the index multikey, the
      IndexBuildInterceptor records the multikey paths only in memory
      (src/mongo/db/index_builds/index_build_interceptor.cpp):

      // No need to take the multikeyPaths mutex if this would not change any multikey state.
      >if (op == Op::kInsert && isMultikey) {
          std::unique_lock<std::mutex> lk(_multikeyPathMutex);
          if (_multikeyPaths) {
              MultikeyPathTracker::mergeMultikeyPaths(&_multikeyPaths.value(), multikeyPaths);
          } else {
              _multikeyPaths = multikeyPaths;
          }
      }
      

      The rows written to the side writes table carry only the operation type
      and the serialized key string — no multikey information:

      toInsert.emplace_back(BSON("op" << (op == Op::kInsert ? "i" : "d") << "key" << binData));
      

      At commit time, the cached in-memory paths are the only thing that marks
      the index multikey for those writes
      (src/mongo/db/index_builds/multi_index_block.cpp, commit()):

      if (auto interceptor = indexCatalogEntry->indexBuildInterceptor()) {
          auto multikeyPaths = interceptor->getMultikeyPaths();
          if (multikeyPaths) {
              indexCatalogEntry->setMultikey(opCtx, collection, {}, multikeyPaths.value());
          }
          ...
      }
      

      Root cause

      The resume state persisted on clean shutdown (added by SERVER-50899)
      captures multikey information only from the bulk builder, i.e. only what
      the collection scan observed
      (MultiIndexBlock::_buildIndexStateInfo()):

      indexStateInfo.setIsMultikey(index.bulk->isMultikey());
      
      std::vector<MultikeyPath> multikeyPaths;
      for (const auto& multikeyPath : index.bulk->getMultikeyPaths()) { ... }
      

      The interceptor's in-memory _multikeyPaths is never persisted. On
      startup the interceptor is reconstructed empty
      (IndexBuildBlock::initForResume() passes
      LazyRecordStore::CreateMode::openExisting, and the constructor has no
      restore path), and re-draining the remaining side writes cannot recover
      the information: applyIndexBuildSideWrite() deserializes one key per
      row and applies it with an empty MultikeyPaths, so

      bool SortedDataIndexAccessMethod::shouldMarkIndexAsMultikey(...) const {
          return numberOfKeys > 1 || isMultikeyFromPaths(multikeyPaths);   // 1 > 1 || false
      }
      

      never fires. At commit, interceptor->getMultikeyPaths() returns none,
      the restored bulk builder reports isMultikey = false, and
      setMultikey() is never called.

      Consequence

      The build commits an index that physically contains multikey entries but
      is flagged non-multikey in the catalog. The query planner then builds
      index bounds and covered plans that are only legal for non-multikey
      indexes:

      • a compound-interval predicate such as find({a: {$gte: 4, $lte: 3)}}
        intersects the bounds to an empty interval and *silently misses matching
        array documents*;
      • covered projections can return an arbitrary array element as the field
        value.

      validate reports the collection as invalid for the affected index, but
      nothing repairs the flag automatically; the incorrect results persist
      until the index is rebuilt.

      Why existing tests do not catch this

      The resumable index build test harness inserts its "side writes" while the
      build hangs at hangBeforeBuildingIndexbefore the collection scan
      — and then calls awaitLastOpCommitted()
      (jstests/noPassthrough/libs/index_builds/index_build.js,
      createIndexesWithSideWrites()). The scan's majority-committed snapshot
      therefore includes those documents, the bulk builder also observes the
      arrays, and its multikey state is what round-trips through the persisted
      IndexStateInfo. The gap — a multikey write landing after the scan —
      is never exercised. The adjacent fix SERVER-127943 restored the *bulk
      builder's* wildcard metadata flag on resume but did not touch the
      interceptor path.

      Affected code

      • Persist: MultiIndexBlock::_buildIndexStateInfo in
        src/mongo/db/index_builds/multi_index_block.cpp
      • In-memory-only tracking: IndexBuildInterceptor::sideWrite /
        getMultikeyPaths in
        src/mongo/db/index_builds/index_build_interceptor.cpp
      • Resume: IndexBuildBlock::initForResume in
        src/mongo/db/index_builds/index_build_block.cpp
      • Drain-time application: SortedDataIndexAccessMethod::applyIndexBuildSideWrite
        in src/mongo/db/index/index_access_method.cpp

      Proposed fix

      In _buildIndexStateInfo(), fold the interceptor's recorded multikey
      paths into the persisted multikey state:

      bool isMultikey = index.bulk->isMultikey();
      MultikeyPaths multikeyPathsToPersist = index.bulk->getMultikeyPaths();
      if (const auto* interceptor = index.block->getIndexBuildInterceptor()) {
          if (auto sideWritesMultikeyPaths = interceptor->getMultikeyPaths()) {
              isMultikey = true;
              if (multikeyPathsToPersist.empty()) {
                  multikeyPathsToPersist = std::move(*sideWritesMultikeyPaths);
              } else if (multikeyPathsToPersist.size() == sideWritesMultikeyPaths->size()) {
                  MultikeyPathTracker::mergeMultikeyPaths(&multikeyPathsToPersist,
                                                          *sideWritesMultikeyPaths);
              }
          }
      }
      indexStateInfo.setIsMultikey(isMultikey);
      

      On resume the merged state is restored into the bulk builder (the existing
      BulkBuilderImpl resume constructor already reads
      stateInfo.getIsMultikey() and stateInfo.getMultikeyPaths()), and
      the existing commit path marks the index multikey. No IDL or on-disk
      format change is required, and the change can only widen the multikey
      state — the conservative, always-safe direction.

      Marking an index multikey when it need not be is safe (the planner is
      merely less aggressive); failing to mark it multikey produces incorrect
      query results.

      Impact and severity

      High. Silent incorrect query results from a successfully committed
      index. The window — a write that makes the index multikey arriving after
      the collection scan, followed by a clean shutdown before the build
      commits — is routinely reachable in production: rolling maintenance
      restarts of replica set members while index builds run on busy
      collections is exactly the scenario resumable index builds were designed
      for.

            Assignee:
            Alex Sarkesian
            Reporter:
            Seid Muhammed (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: