-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 6.0.0, 7.0.0, 8.0.0
-
Component/s: Index Maintenance
-
None
-
Environment:Replica set members undergoing clean restarts (e.g. rolling maintenance,
package upgrades) while hybrid index builds are in progress on
collections receiving concurrent writes that introduce arrays into
indexed fields.
-
Storage Execution
-
ALL
-
-
Storage Execution 2026-08-03
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
A hybrid index build indexes documents through two channels: the
collection scan feeds the bulk builder, and concurrent client writes are
intercepted and buffered in the side writes table, which is drained into
the index before commit.
When a concurrent write makes the index multikey, the
IndexBuildInterceptor records the multikey paths only in memory
(src/mongo/db/index_builds/index_build_interceptor.cpp):
// No need to take the multikeyPaths mutex if this would not change any multikey state. >if (op == Op::kInsert && isMultikey) { std::unique_lock<std::mutex> lk(_multikeyPathMutex); if (_multikeyPaths) { MultikeyPathTracker::mergeMultikeyPaths(&_multikeyPaths.value(), multikeyPaths); } else { _multikeyPaths = multikeyPaths; } }
The rows written to the side writes table carry only the operation type
and the serialized key string — no multikey information:
toInsert.emplace_back(BSON("op" << (op == Op::kInsert ? "i" : "d") << "key" << binData));
At commit time, the cached in-memory paths are the only thing that marks
the index multikey for those writes
(src/mongo/db/index_builds/multi_index_block.cpp, commit()):
if (auto interceptor = indexCatalogEntry->indexBuildInterceptor()) { auto multikeyPaths = interceptor->getMultikeyPaths(); if (multikeyPaths) { indexCatalogEntry->setMultikey(opCtx, collection, {}, multikeyPaths.value()); } ... }
Root cause
The resume state persisted on clean shutdown (added by SERVER-50899)
captures multikey information only from the bulk builder, i.e. only what
the collection scan observed
(MultiIndexBlock::_buildIndexStateInfo()):
indexStateInfo.setIsMultikey(index.bulk->isMultikey()); std::vector<MultikeyPath> multikeyPaths; for (const auto& multikeyPath : index.bulk->getMultikeyPaths()) { ... }
The interceptor's in-memory _multikeyPaths is never persisted. On
startup the interceptor is reconstructed empty
(IndexBuildBlock::initForResume() passes
LazyRecordStore::CreateMode::openExisting, and the constructor has no
restore path), and re-draining the remaining side writes cannot recover
the information: applyIndexBuildSideWrite() deserializes one key per
row and applies it with an empty MultikeyPaths, so
bool SortedDataIndexAccessMethod::shouldMarkIndexAsMultikey(...) const { return numberOfKeys > 1 || isMultikeyFromPaths(multikeyPaths); // 1 > 1 || false }
never fires. At commit, interceptor->getMultikeyPaths() returns none,
the restored bulk builder reports isMultikey = false, and
setMultikey() is never called.
Consequence
The build commits an index that physically contains multikey entries but
is flagged non-multikey in the catalog. The query planner then builds
index bounds and covered plans that are only legal for non-multikey
indexes:
- a compound-interval predicate such as find({a: {$gte: 4, $lte: 3)}}
intersects the bounds to an empty interval and *silently misses matching
array documents*; - covered projections can return an arbitrary array element as the field
value.
validate reports the collection as invalid for the affected index, but
nothing repairs the flag automatically; the incorrect results persist
until the index is rebuilt.
Why existing tests do not catch this
The resumable index build test harness inserts its "side writes" while the
build hangs at hangBeforeBuildingIndex — before the collection scan
— and then calls awaitLastOpCommitted()
(jstests/noPassthrough/libs/index_builds/index_build.js,
createIndexesWithSideWrites()). The scan's majority-committed snapshot
therefore includes those documents, the bulk builder also observes the
arrays, and its multikey state is what round-trips through the persisted
IndexStateInfo. The gap — a multikey write landing after the scan —
is never exercised. The adjacent fix SERVER-127943 restored the *bulk
builder's* wildcard metadata flag on resume but did not touch the
interceptor path.
Affected code
- Persist: MultiIndexBlock::_buildIndexStateInfo in
src/mongo/db/index_builds/multi_index_block.cpp - In-memory-only tracking: IndexBuildInterceptor::sideWrite /
getMultikeyPaths in
src/mongo/db/index_builds/index_build_interceptor.cpp - Resume: IndexBuildBlock::initForResume in
src/mongo/db/index_builds/index_build_block.cpp - Drain-time application: SortedDataIndexAccessMethod::applyIndexBuildSideWrite
in src/mongo/db/index/index_access_method.cpp
Proposed fix
In _buildIndexStateInfo(), fold the interceptor's recorded multikey
paths into the persisted multikey state:
bool isMultikey = index.bulk->isMultikey(); MultikeyPaths multikeyPathsToPersist = index.bulk->getMultikeyPaths(); if (const auto* interceptor = index.block->getIndexBuildInterceptor()) { if (auto sideWritesMultikeyPaths = interceptor->getMultikeyPaths()) { isMultikey = true; if (multikeyPathsToPersist.empty()) { multikeyPathsToPersist = std::move(*sideWritesMultikeyPaths); } else if (multikeyPathsToPersist.size() == sideWritesMultikeyPaths->size()) { MultikeyPathTracker::mergeMultikeyPaths(&multikeyPathsToPersist, *sideWritesMultikeyPaths); } } } indexStateInfo.setIsMultikey(isMultikey);
On resume the merged state is restored into the bulk builder (the existing
BulkBuilderImpl resume constructor already reads
stateInfo.getIsMultikey() and stateInfo.getMultikeyPaths()), and
the existing commit path marks the index multikey. No IDL or on-disk
format change is required, and the change can only widen the multikey
state — the conservative, always-safe direction.
Marking an index multikey when it need not be is safe (the planner is
merely less aggressive); failing to mark it multikey produces incorrect
query results.
Impact and severity
High. Silent incorrect query results from a successfully committed
index. The window — a write that makes the index multikey arriving after
the collection scan, followed by a clean shutdown before the build
commits — is routinely reachable in production: rolling maintenance
restarts of replica set members while index builds run on busy
collections is exactly the scenario resumable index builds were designed
for.
- is related to
-
SERVER-50899 Keep track of multikey in resumable index build state
-
- Closed
-
-
SERVER-127943 Resumable primary-driven build of a multikey wildcard index crashes the secondary with WT_DUPLICATE_KEY
-
- Closed
-