Add jstest pinning $merge bit-identical replay idempotency across whenMatched modes

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      $merge is documented as idempotent under replay for the three whenMatched modes that touch existing target rows: replace, merge, and keepExisting. Running the same deterministic aggregation pipeline twice against the same source collection must produce the same target-collection state. This property is the load-bearing replay-safety contract that allows $merge to be used as the primitive behind materialized-view rematerialization and exactly-once stream-processing patterns.

      Today this invariant is exercised implicitly across many specific tests in jstests/aggregation/sources/merge/ (one per mode + edge case). It is not exercised by a single test that holds the source fixed, replays the same pipeline twice, and asserts the target state is bit-identical between the two runs across the three modes side by side.

      This ticket proposes a single new jstest that pins exactly that invariant.

      Scope

      • One new file: jstests/aggregation/sources/merge/concurrent_replay_idempotency.js (~158 lines).
      • No source changes; no helpers changed.
      • Imports: dropWithoutImplicitRecreate from jstests/aggregation/extras/merge_helpers.js, orderedArrayEq from jstests/aggregation/extras/utils.js — matching neighboring tests in the same directory.

      Test shape

      Three IIFEs (one per whenMatched mode: replace, merge, keepExisting). Each:

      1. Drop and recreate a deterministic 1000-event source collection.
      2. Drop and recreate a target collection.
      3. Run an aggregation pipeline that groups events into 10 buckets and $merge}}s into the target with {{whenNotMatched: "insert".
      4. Snapshot the target sorted by _id; compute a deterministic fingerprint (sum of grouped values + doc count).
      5. Re-run the exact same pipeline on the exact same source.
      6. Snapshot again; assert orderedArrayEq(snap1, snap2) AND fingerprint1 === fingerprint2.

      What this catches

      A future patch that breaks $merge's bit-identical replay — e.g.:

      • whenMatched semantics regression (mode falls through wrong path on a replay)
      • upsert misordering under concurrent shard targets
      • nondeterministic _id generation on a path that should be upsert-keyed

      …will fail one of the two assertions.

      Verification

      • node --input-type=module --check jstests/aggregation/sources/merge/concurrent_replay_idempotency.js clean.
      • Parity with jstests/aggregation/sources/merge/mode_keep_existing_insert.js style verified.
      • Test is green-only; no counterexample mode.

      Acceptance criteria

      • File lands under jstests/aggregation/sources/merge/.
      • Test passes under standard resmoke invocation against an unsharded mongod.
      • All three whenMatched modes (replace, merge, keepExisting) exercised.
      • Imports + style match neighboring test files.

      Related work

      • Existing per-mode tests under jstests/aggregation/sources/merge/mode_*.js cover each mode in isolation; this test cross-cuts them under a single replay invariant.
      • Adjacent context: SERVER-99827 explicitly excludes $merge tests from config-transition suites due to unsafe retries. The invariant pinned here ($merge replay = bit-identical) is the structural property that has to hold before any of those exclusions can be safely reverted. This ticket is a step toward that, not a fix for SERVER-99827.

      Implementation status

      Draft authored and node --check verified locally. Forbidden-term review complete. PR to follow on this ticket.

            Assignee:
            Unassigned
            Reporter:
            Mehar Grewal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: