• Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Optimization
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Found by an AI bug hunt and confirmed by a differential oracle.

      Problem

      When a $sort followed by a $group whose _id is a single-field object (one field v taken from $a) is rewritten to a DISTINCT_SCAN, the group key for a document that has no a comes back as an empty document instead of a document with v set to null. $group treats a missing key value as null (SERVER-110423), and every other plan returns v set to null: a collection scan, a non-indexed collection, or the same group with _id: "$a" on the distinct scan. The classic and SBE engines return the same wrong key, so an engine comparison does not show it.

      Repro

      Collection c with an index on a: 1, b: 1; document _id: 2 has no a:

      db.c.insertMany([{_id: 1, a: 1, b: 1, d: "x"}, {_id: 2, b: 2, d: "y"}, {_id: 3, a: 2, b: 3, d: "z"}]);
      db.c.createIndex({a: 1, b: 1});
      db.c.aggregate([{$sort: {a: 1, b: 1}}, {$group: {_id: {v: "$a"}, f: {$first: "$d"}}}]);
      

      Group result for the document without a, by plan:

      DISTINCT_SCAN, SBE or classic (default):   { _id: { },        f: "y" }   <-- wrong
      collection scan (hint {$natural: 1}):      { _id: { v: null }, f: "y" }
      DISTINCT_SCAN with _id: "$a":              { _id: null,        f: "y" }
      no index, classic, optimization barriers:  { _id: { v: null }, f: "y" }
      

      The key also depends on which document the scan reaches first. With one document that has a: null, b: 2 and one that has no a and b: 1 (null and missing fall in one group, $first: "$_id"), the distinct scan returns the empty-document key, because the first index entry is the document without a. With the b values swapped it returns the key with v set to null. A collection scan returns the key with v set to null in both orders.

      Self-contained jstest (noPassthrough style); fails at its last assertions after its controls and mechanism checks pass:

      /**
       * {$sort: {a: 1}} followed by {$group: {_id: {v: "$a"}}} is rewritten to a DISTINCT_SCAN when an
       * index on 'a' exists. For a document that has no 'a' the group key comes back as {} instead of
       * {v: null}: the rewrite builds the key with an object expression, which omits a missing value,
       * while $group (collection scan, any other plan, or _id: "$a") turns a missing key value into
       * null. The same wrong answer comes from the classic and SBE engines.
       */
      const conn = MongoRunner.runMongod({setParameter: {internalQueryDisablePlanCache: true}});
      const db = conn.getDB(jsTestName());
      assert.commandWorked(db.c.insertMany([
          {_id: 1, a: 1, b: 1, d: "x"},
          {_id: 2, b: 2, d: "y"},  // no 'a'
          {_id: 3, a: 2, b: 3, d: "z"},
      ]));
      assert.commandWorked(db.c.createIndex({a: 1, b: 1}));
      
      const sorted = (docs) => docs.map((d) => tojson(d)).sort();
      function stages(explain) {
          const names = new Set();
          (function walk(x) {
              if (Array.isArray(x)) return x.forEach(walk);
              if (x === null || typeof x !== "object") return;
              if (typeof x.stage === "string") names.add(x.stage);
              Object.keys(x).filter((k) => k.startsWith("$")).forEach((k) => names.add(k));
              Object.values(x).forEach(walk);
          })(explain);
          return names;
      }
      function run(label, engine, pipeline, options = {}) {
          assert.commandWorked(
              db.adminCommand({setParameter: 1, internalQueryFrameworkControl: engine}));
          const used = stages(db.c.explain().aggregate(pipeline, options));
          const got = sorted(db.c.aggregate(pipeline, options).toArray());
          jsTest.log.info("run", {label, engine, distinctScan: used.has("DISTINCT_SCAN"), got});
          return {got, distinctScan: used.has("DISTINCT_SCAN")};
      }
      
      const objKey = [{$sort: {a: 1, b: 1}}, {$group: {_id: {v: "$a"}, f: {$first: "$d"}}}];
      const pathKey = [{$sort: {a: 1, b: 1}}, {$group: {_id: "$a", f: {$first: "$d"}}}];
      const wantObj = sorted([{_id: {v: null}, f: "y"}, {_id: {v: 1}, f: "x"}, {_id: {v: 2}, f: "z"}]);
      const wantPath = sorted([{_id: null, f: "y"}, {_id: 1, f: "x"}, {_id: 2, f: "z"}]);
      
      // Controls: a collection scan and the plain-path key give null for the missing value.
      assert.eq(run("collscan", "trySbeEngine", objKey, {hint: {$natural: 1}}).got, wantObj, "collscan");
      const path = run("path key on distinct scan", "trySbeEngine", pathKey);
      assert(path.distinctScan, "the path-key control did not use a DISTINCT_SCAN");
      assert.eq(path.got, wantPath, "_id: '$a' on a distinct scan");
      // Mechanism: the object-key pipeline uses the distinct scan.
      for (const engine of ["trySbeEngine", "forceClassicEngine"]) {
          assert(run("object key", engine, objKey).distinctScan, "no DISTINCT_SCAN: the repro tests nothing");
      }
      // Fails at the local build (2026-09-17): the key for the document without 'a' is {} in both engines.
      assert.eq(run("object key", "forceClassicEngine", objKey).got, wantObj, "object key, classic");
      assert.eq(run("object key", "trySbeEngine", objKey).got, wantObj, "object key, SBE");
      MongoRunner.stopMongod(conn);
      

      Cause (from code reading, not verified)

      rewriteGroupAsTransformOnFirstDocument in src/mongo/db/pipeline/document_source_group_base.cpp builds the key for a single-field object _id with ExpressionObject::create, which omits a field whose value is missing. GroupFromFirstDocumentTransformation::applyTransformation in group_from_first_document_transformation.cpp replaces a missing value with null only when the whole expression value is missing, so a missing v is left out and the key stays an empty document. The classic $group path computes the key from the field expression and null-fills it (GroupProcessorBase::computeGroupKey), then wraps it in the object.

      Impact

      Silent wrong group keys under default settings for $sort + $group with $first/$last (or no accumulators) over an index, when the _id is a single-field object and a document has no value for that field. Downstream stages and $out/$merge see an empty _id (and $_id.v as missing) instead of null, and the result differs from the same query without the index. When both null and missing values exist, the key of the combined group depends on index order.

      Related

      • SERVER-21992: inconsistent results when grouping by possibly missing values (open); jstests/aggregation/sources/group/groupMissing.js marks the single-field object case as a known bug there and does not cover $sort + $group over an index.
      • SERVER-60977: closed; states that a single-field document _id conflates null and missing.
      • SERVER-110423, SERVER-111486: closed; distinct scan on a sparse index hid null/missing groups (rows dropped, a different shape).

      Which behavior is intended for a missing value in a single-field object _id is for the owners to decide (SERVER-21992 leaves the target open). The distinct-scan rewrite currently disagrees with the other plans of the same query, and with itself depending on index order.

            Assignee:
            Unassigned
            Reporter:
            Zixuan Zhuang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: