• Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Optimization
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Found by an AI bug hunt and confirmed by a differential oracle: results differ with internalQueryPlannerPushdownFilterToIxscanForSort on and off, while the classic and SBE engines agree with each other.

      Problem

      A $expr filter that compares a field with null using $lt, on a field that is not the leading field of an index that provides the sort, is pushed into a full index scan. In aggregation expressions a missing field sorts below null, so the comparison is true for documents without the field. The index key for a missing field is null, and null is not less than null, so those documents are dropped before the fetch can apply the exact $expr. Both engines return the same wrong result. The knob internalQueryPlannerPushdownFilterToIxscanForSort is on by default.

      Repro

      Fails at the local build of 2026-09-17; no later commit changes the filter pushdown. The index is on c then w. The query is $expr with $lt of $w and null, sorted by c, projecting _id. Documents 1 and 5 have no w, document 2 has w null, and documents 3 and 4 have a number and a string.

      Run Result (expected _id 1, 5)
      pushdown on, classic engine no documents
      pushdown on, SBE no documents
      pushdown off, either engine 1, 5
      collection scan ($natural hint) 1, 5

      With the pushdown on, the IXSCAN stage in the explain carries the filter shown below. Self-contained jstest; controls first, the failing assertion last:

      // IXSCAN filter in the explain with the pushdown on: { "w" : { "$_internalExprLt" : null } }
      /**
       * A {$expr: {$lt: [<field>, null]}} filter pushed into a full index scan that exists only to
       * provide a sort drops documents where the field is missing: the index key for a missing field is
       * null and null < null is false, while in aggregation expressions missing sorts below null.
       * The result is the same wrong answer in the classic and SBE engines.
       */
      const conn = MongoRunner.runMongod({setParameter: {internalQueryDisablePlanCache: true}});
      const db = conn.getDB(jsTestName());
      assert.commandWorked(db.c.insertMany([
          {_id: 1, c: 1},
          {_id: 2, c: 2, w: null},
          {_id: 3, c: 3, w: 1},
          {_id: 4, c: 4, w: "x"},
          {_id: 5, c: 5},
      ]));
      assert.commandWorked(db.c.createIndex({c: 1, w: 1}));
      
      const pipeline = [{$match: {$expr: {$lt: ["$w", null]}}}, {$sort: {c: 1}}, {$project: {_id: 1}}];
      const ids = (docs) => docs.map((d) => d._id);
      function ixscanFilters(explain) {
          const found = [];
          (function walk(x) {
              if (Array.isArray(x)) return x.forEach(walk);
              if (x === null || typeof x !== "object") return;
              if (x.stage === "IXSCAN") found.push(x.filter);
              Object.values(x).forEach(walk);
          })(explain);
          return found;
      }
      function run(label, engine, pushdown, hint) {
          assert.commandWorked(db.adminCommand({
              setParameter: 1,
              internalQueryFrameworkControl: engine,
              internalQueryPlannerPushdownFilterToIxscanForSort: pushdown,
          }));
          const opts = hint ? {hint} : {};
          const filters = ixscanFilters(db.c.explain().aggregate(pipeline, opts));
          const got = ids(db.c.aggregate(pipeline, opts).toArray());
          jsTest.log.info("run", {label, engine, pushdown, hint, ixscanFilters: filters, got});
          return {got, filters};
      }
      
      // Controls: missing < null is true, so documents 1 and 5 match, when the filter is not pushed
      // into the index scan or when the query scans the collection.
      assert.eq(run("pushdown off", "forceClassicEngine", false).got, [1, 5], "pushdown off, classic");
      assert.eq(run("pushdown off", "trySbeEngine", false).got, [1, 5], "pushdown off, SBE");
      assert.eq(run("collection scan", "forceClassicEngine", true, {$natural: 1}).got, [1, 5], "collscan");
      // Mechanism: with the pushdown on, the filter is in the index scan.
      assert(run("pushdown on", "forceClassicEngine", true).filters.some((f) => f !== undefined),
             "the filter was not pushed into the index scan; the repro does not test the bug");
      // Fails at the local build (2026-09-17): returns [] in both engines.
      assert.eq(run("pushdown on", "forceClassicEngine", true).got, [1, 5], "pushdown on, classic");
      assert.eq(run("pushdown on", "trySbeEngine", true).got, [1, 5], "pushdown on, SBE");
      MongoRunner.stopMongod(conn);
      

      Cause (from code reading, not verified)

      • rewrite_expr.cpp accepts a null right-hand side, so the $expr becomes an internal comparison.
      • The pushdown (pushdownFilterToFullIxscan() in src/mongo/db/query/planner_access.cpp) treats only $exists and $type: null as unsafe for the index scan filter.
      • For bounded scans index_bounds_builder.cpp widens a null comparison to include missing values and marks the predicate as needing a fetch; the pushdown path has no equivalent guard.

      Impact

      Silent wrong results with default settings for an $expr null comparison combined with a sort that an index provides.

      Related

      • SERVER-103601: introduced the pushdown for full index scans.
      • SERVER-125720: an earlier bug in the same knob ($expr with dotted paths), found by a property test. index_for_sort_pbt.js avoids missing fields.

            Assignee:
            Unassigned
            Reporter:
            Zixuan Zhuang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: