ExportXMLWordPrintableJSON

    • DB Integration & Observability
    • Fully Compatible
    • QE 2026-09-28
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This change fixes flakiness in the test `jstests/noPassthrough/query/change_streams/change_stream_pre_image_startup_recovery.js`.

      Previously the test could non-deterministically fail with errors such as ([example failed run](https://parsley.corp.mongodb.com/test/mongodb_mongo_master_enterprise_amazon_linux2023_arm64_all_feature_flags_rbe___jstests_noPassthrough:no_passthrough_patch_94c253e337508c29d836607582fdac497aada7ec_6ab5014f17a7e50007bab825_26_09_24_10_54_24/0/c52c309ba0a9c9135f0312f504a748df?bookmarks=0%2C6842&selectedLineRange=L6755&shareLine=6755)):
      ```
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] ✘ change stream pre-image truncation persists across crash + recovery > primary unclean shutdown with all pre-images expired
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] expected [ ] to equal [ [Object], [Object], [Object], [Object] ]
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + expected - actual
      [js_test:change_stream_pre_image_startup_recovery] [jsTest]
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] -[ ]
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] +[
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + {
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "nsUUID" : UUID("3751d912-ca36-4846-a5cc-0d45996f4ff2"), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "ts" : Timestamp(1790248367, 8), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "applyOpsIndex" : NumberLong(0) [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      ,
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "operationTime" : ISODate("2026-09-24T11:12:47.725Z"),
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "preImage" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" : 0, [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "v" : 0 [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + },
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + {
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "nsUUID" : UUID("3751d912-ca36-4846-a5cc-0d45996f4ff2"), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "ts" : Timestamp(1790248367, 9), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "applyOpsIndex" : NumberLong(0) [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      ,
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "operationTime" : ISODate("2026-09-24T11:12:47.729Z"),
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "preImage" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" : 0, [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "v" : 1 [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + },
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + {
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "nsUUID" : UUID("3751d912-ca36-4846-a5cc-0d45996f4ff2"), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "ts" : Timestamp(1790248367, 10), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "applyOpsIndex" : NumberLong(0) [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      ,
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "operationTime" : ISODate("2026-09-24T11:12:47.732Z"),
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "preImage" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" : 0, [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "v" : 2 [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + },
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + {
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "nsUUID" : UUID("3751d912-ca36-4846-a5cc-0d45996f4ff2"), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "ts" : Timestamp(1790248367, 11), [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "applyOpsIndex" : NumberLong(0) [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      ,
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "operationTime" : ISODate("2026-09-24T11:12:47.736Z"),
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "preImage" :

      { [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "_id" : 0, [js_test:change_stream_pre_image_startup_recovery] [jsTest] + "v" : 3 [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }

      [js_test:change_stream_pre_image_startup_recovery] [jsTest] + }
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] +]
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] : pre-images changed after recovery
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] _getErrorWithCode@src/mongo/shell/utils.js:32:13
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] doassert@src/mongo/shell/assert.js:48:14
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] _doassert@src/mongo/shell/assert.js:182:13
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] assert.eq@src/mongo/shell/assert.js:310:14
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] jstests/noPassthrough/query/change_streams/change_stream_pre_image_startup_recovery.js:429:16
      [js_test:change_stream_pre_image_startup_recovery] [jsTest] run@/data/mci/3aa4893211db07a24d5b51376b1fc59e/tmp/bazel-utput-root/ed0f11038c63cabc9b66e8e9998680c3/execroot/_main/bazel-out/aarch64-opt/bin/jstests/libs/mochalite.js:375:31
      ```

      The failure happens in a primary unclean shutdown with all pre-images expired [here](https://github.com/10gen/mongo/blob/fc72883d242187c1bbbfdf135b973ba2efa37b3e/jstests/noPassthrough/query/change_streams/change_stream_pre_image_startup_recovery.js#L429).

      Before the primary gets killed, it had truncated all pre-images in memory. After the crash and restart, the restarted node read back the 4 pre-images, but expected to find 0.
      That means the truncation that existed in memory on the primary before it is crashed was completely lost across recovery.

      With `featureFlagUseReplicatedTruncatesForDeletions` enabled (as in this run), `cleanupPreImagesCollectionAfterUncleanShutdown()` exits early and the recovery is supposed to get the truncation back via replication.

      The primary's fsync checkpoint is before the truncation operation, then the checkpoint thread is paused. Then the truncation happens, and it is only in the primary's own oplog.
      On unclean restart, the recovery truncates everything after the stable timestamp and removes unapplied oplog entries after it. The crashed node discards its own post-stable oplog, so it can never replay its own truncate that it executed after the stable timestamp.
      The only way the truncation survives is if the other node had already applied it and then the restarted node re-fetches it from the survivor.
      This made the test inherently racy.

      This PR makes the test deterministic by adding `awaitReplication()` at various points in the test.

            Assignee:
            Jan Steemann
            Reporter:
            Jan Steemann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: