-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
ALL
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary
When an aggregation is eligible for a covered index plan, the covered projection reconstructs documents from index keys and materializes an indexed-but-absent path as an explicit null. A later $match on $exists: false then fails to match a document that genuinely lacks the field, so the pipeline returns wrong results.
The symptom is that appending a $count to a pipeline changes how many documents the preceding $match matched. $count itself is not special — what matters is that it (and anything else that needs no document fields) lets the query planner see that nothing downstream needs the full document. That dependency analysis is what makes a covered plan eligible in the first place; see Root Cause for the mechanism.
Steps to reproduce
const coll = db.c; coll.drop(); assert.commandWorked(coll.insert({_id: 5})); // 'obj.num' is absent assert.commandWorked(coll.createIndex({"_id": -1, "obj.num": 1})); const match = [{$sort: {_id: 1}}, {$limit: 1}, {$match: {"obj.num": {$not: {$exists: true}}}}]; // The $match alone matches the document: assert.eq(1, coll.aggregate(match).itcount()); // passes // Appending $count changes the answer: assert.eq([{num: 1}], coll.aggregate(match.concat([{$count: "num"}])).toArray());
Expected: [\{num: 1}]
Actual: []
Root cause
Without $count, $match is the pipeline's last stage, so its output is the query result — every field of the matched document must be returned to the caller, forcing a real FETCH:
pipeline: [$sort, $limit, $match]
stages : $cursor, $match
plan : LIMIT <- FETCH <- IXSCAN
result : [ { "_id" : 5 } ] <- correct
With $count appended, the pipeline's dependency analysis determines that nothing downstream needs the document at all beyond what $match itself reads (obj.num) — $count only needs to know the document existed, not its contents. Since the index already contains everything $match touches, the planner can skip the FETCH entirely and serve the query straight from the
index:
pipeline: [$sort, $limit, $match, $count]
stages : $cursor, $match, $group, $project
plan : LIMIT <- PROJECTION_DEFAULT <- IXSCAN (no FETCH)
parsedQuery: {}
result : [ ] <- WRONG
Note $match is not pushed into the $cursor's own filter in either case (parsedQuery: {} both times) — it runs as a separate pipeline stage after $cursor. The only thing $count changes is whether the $cursor needs to return full documents or can get away with an index-only (covered) projection. Any stage combination with the same property — nothing downstream needs fields beyond what an existing $match reads — should trigger the same bug; $count is just the simplest example.
The covered projection invents the absent path as null:
coll.find({}, {_id: 1, "obj.num": 1}).hint({"_id": -1, "obj.num": 1})
-> [ { "_id" : 5, "obj" : { "num" : null } } ] <- fabricated
coll.find({}, {_id: 1, "obj.num": 1}).hint({$natural: 1})
-> [ { "_id" : 5 } ] <- correct
An index key cannot distinguish a missing field from an explicit null, so a covered plan must not be used when a consumer can observe that difference. Historically this was tolerable because the caller only saw the projected output; here the fabricated null is fed to a downstream $match and silently changes query results.
Suggested fix
Either:
- do not choose a covered plan when a downstream stage depends on field existence (a $match with $exists / $type / null-equality on a covered path); or
- have the covered projection omit absent paths rather than emitting null.
The first is narrower; the second risks changing long-documented covered-query output shape and probably needs its own discussion.
Notes
Found by aggregation_optimization_fuzzer (BF-46309). Deterministic; confirmed
by the fuzzer's own minimizer (stats.deterministic: true), which reduced the original 200 documents / 63 indexes / 2460 pipelines to one of each. Not a regression at the BF's first-failing revision — that commit only adds argument-count assertions to _resultSetsEqualUnordered.
The original BF's failure message frames this as an disableMatchExpressionOptimization/disablePipelineOptimization on/off difference between two servers. That framing does not reproduce: across 70 independent local trials (both failpoint settings; 40+ freshly started mongods; debug and release builds; default and 1GB-capped WT cache), every configuration returns the wrong answer on both settings — never the "correct" side CI reported.
The $cursor's own parsedQuery is {} either way, so there's no match-expression rewriting for those failpoints to affect in this plan shape. Treat the on/off split in the original report as incidental to environment (most likely a plan-selection difference between the two CI hosts/processes), not a required trigger — the bug reproduces unconditionally once a covered plan is chosen.
Confirmed on both master and v8.0 (ea0ab02). v7.0/v8.2/v8.3 not checked, but the mechanism isn't version-specific.
- duplicates
-
SERVER-133440 Covered plans can be incorrectly generated for sort queries even without projections
-
- In Code Review
-