startAfter fails with NoMatchingDocument on the resume-token event: the "required" pre-image check is applied to the event the stream must start after

ExportXMLWordPrintableJSON

    • DB Integration & Observability
    • Fully Compatible
    • ALL
    • v9.0
    • Hide

      Single-node replica set, mongosh.

      const c = db.getSiblingDB("repro").items;
c.drop();
c.insertOne({ _id: 1, v: 0 });

// 1. Pre-images are OFF. Make one update and keep its resume token.
const s0 = c.watch();
c.updateOne({ _id: 1 }, { $set: { v: 1 } });
let e; while (!(e = s0.tryNext())) {}
const tokenOfOldUpdate = e._id;
s0.close();

// 2. Enable pre- and post-images. Every later change has both images.
db.getSiblingDB("repro").runCommand(
  { collMod: "items", changeStreamPreAndPostImages: { enabled: true } });
c.updateOne({ _id: 1 }, { $set: { v: 2 } });

// 3. Resume AFTER the old update. The only event to return is the new update.
c.watch([], { startAfter: tokenOfOldUpdate,
              fullDocumentBeforeChange: "required" }).tryNext();

      Actual result (7.0.37, 8.0.26, 8.2.11)

      startAfter,  fullDocumentBeforeChange=whenAvailable: OK ["update v=1->2"]
startAfter,  fullDocumentBeforeChange=required     : ERROR 47 NoMatchingDocument -
  Change stream was configured to require a pre-image for all update, delete and
  replace events, but the pre-image was not found for event: {operationType: ...
resumeAfter, fullDocumentBeforeChange=required     : ERROR 47 NoMatchingDocument (same)
startAfter,  fullDocument=required                 : ERROR 47 NoMatchingDocument -
  Change stream was configured to require a post-image for all update events, ...

      The whenAvailable line shows that the new update is the only event after the token and that its pre-image exists (v=1).

      Expected result

      The stream opens and returns the update v=1->2 with its pre-image.

      Show
      Single-node replica set, mongosh. const c = db.getSiblingDB( "repro" ).items; c.drop(); c.insertOne({ _id: 1, v: 0 }); // 1. Pre-images are OFF. Make one update and keep its resume token. const s0 = c.watch(); c.updateOne({ _id: 1 }, { $set: { v: 1 } }); let e; while (!(e = s0.tryNext())) {} const tokenOfOldUpdate = e._id; s0.close(); // 2. Enable pre- and post-images. Every later change has both images. db.getSiblingDB( "repro" ).runCommand( { collMod: "items" , changeStreamPreAndPostImages: { enabled: true } }); c.updateOne({ _id: 1 }, { $set: { v: 2 } }); // 3. Resume AFTER the old update. The only event to return is the new update. c.watch([], { startAfter: tokenOfOldUpdate, fullDocumentBeforeChange: "required" }).tryNext(); Actual result (7.0.37, 8.0.26, 8.2.11) startAfter, fullDocumentBeforeChange=whenAvailable: OK ["update v=1->2"] startAfter, fullDocumentBeforeChange=required : ERROR 47 NoMatchingDocument - Change stream was configured to require a pre-image for all update, delete and replace events, but the pre-image was not found for event: {operationType: ... resumeAfter, fullDocumentBeforeChange=required : ERROR 47 NoMatchingDocument (same) startAfter, fullDocument=required : ERROR 47 NoMatchingDocument - Change stream was configured to require a post-image for all update events, ... The whenAvailable line shows that the new update is the only event after the token and that its pre-image exists (v=1). Expected result The stream opens and returns the update v=1->2 with its pre-image.
    • QE 2026-10-26
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      With fullDocumentBeforeChange: "required" (or fullDocument: "required"), a stream resumed with startAfter or resumeAfter applies the "required" check to the event the resume token points to. That event must not be read at all: the stream is supposed to continue after it, and it is never returned to the client.

      If that event is an update, replace or delete made before pre-images were enabled, the aggregate fails with NoMatchingDocument (47) before returning anything, although every event after the token has its pre-image. Retrying does not help, the token stays the same.

      Where

      In DocumentSourceChangeStream::_buildPipeline (document_source_change_stream.cpp, v8.0 lines 317-382) the image stages run before the stage that discards the token event:

      DSCSCheckResumability
DSCSAddPreImage                 <- uasserts on "required" for the token event
DSCSAddPostImage
DSCSEnsureResumeTokenPresent    <- discards the token event

      The oplog scan starts at the token's clusterTime inclusive and DSCSCheckResumability passes the token event on, so that the last stage can find it. A token that points to an insert does not fail, an insert needs no pre-image.

      Suggested fix

      Do not apply the "required" check to the event that matches the client's resume token.

      Use case

      We keep resume tokens and change a setting: pre-images are enabled on the collection and the stream is switched to "required". We resume from the stored token because we must continue where the previous version stopped. We expect to just continue, since every event after the token has its pre-image. Instead the stream cannot start: after the restart it stays on the previous token and fails on it. The only ways out are to drop the token (losing the position) or to go back to "whenAvailable".

        1. repro.js
          2 kB

            Assignee:
            Jan Steemann
            Reporter:
            reelo adead (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: