Problem

On a mongos, a resumed $changeStream can return a postBatchResumeToken (PBRT) that sorts strictly before the event resume token the client resumed from. The V2 topology-change stage seeds the router's high water mark by taking the client's event resume token, discarding everything except clusterTime, and re-wrapping that bare Timestamp as a high-water-mark token at the same clusterTime. Because ResumeTokenData::kHighWaterMarkToken (0) sorts before kEventToken (128) at equal clusterTime, the resulting token is a resume-token regression.

This is a change stream resumability correctness bug: a client that persists the PBRT and later resumes from it will rewind and re-deliver events it has already seen.

Note: this is the same symptom as SERVER-121636 (closed as Fixed). The commit that closed SERVER-121636 is what reintroduces it — see "Why this is new in v9" below.

Observed behaviour

Seen in the wild by mongosync's embedded verifier against a v9 sharded cluster. Initial aggregate against admin with allChangesForCluster: true and a resumeAfter event token:

resumeAfter: {"_data": "826AA1FD7F000000012B042C0100296E5A1004B686C982D65E46C2987B35136A2C7290463C..."}

returned firstBatch: [] and:

postBatchResumeToken: {"_data": "826AA1FD7F000000012B0429296E1404"}

Decoding both:

• resumeAfter – clusterTime 0x6AA1FD7F:1, version 2, tokenType 128 (event), txnOpIndex 0, uuid + eventIdentifier
• returned PBRT – clusterTime 0x6AA1FD7F:1, version 2, tokenType 0 (high water mark), and nothing else

Same clusterTime, but the high-water-mark form sorts before the event form. The difference in the _data strings is exactly the tokenType field: 2C0100 (128) versus 29 (0). Note also that operationTime in the reply was 1789001096, nine seconds later than the PBRT's timestamp, so the PBRT was not tracking the oplog's actual latest observed timestamp – it was echoing the resume timestamp in degraded form.

Root cause

src/mongo/db/exec/agg/change_stream_handle_topology_change_v2_stage.cpp:87, in V2StageCursorManager::initialize():

_initializationResumeToken = ResumeToken(resumeTokenData);
LOGV2_DEBUG(12163604, 5, "Using change stream resume token", ...);
setHighWaterMark(_initializationResumeToken.getClusterTime());   // <-- line 87

which lands in the same file at line 252:

void setHighWaterMark(Timestamp highWaterMark) override {
    _mergeCursors->setHighWaterMark(ResumeToken::makeHighWaterMarkToken(
                                        highWaterMark, ResumeTokenData::kDefaultTokenVersion)
                                        .toDocument()
                                        .toBson());
}

The conversion to a bare Timestamp loses tokenType, txnOpIndex, uuid and eventIdentifier.

With an empty first batch nothing ever advances the watermark before the reply is built, so the degraded token is returned verbatim:

• cluster_aggregation_planner.cpp -> ClusterClientCursor::getPostBatchResumeToken()
• -> RouterStagePipeline::getPostBatchResumeToken() (src/mongo/s/query/exec/router_stage_pipeline.cpp:76)
• -> MergeCursorsStage::getHighWaterMark() -> AsyncResultsMerger::getHighWaterMark() (src/mongo/s/query/exec/async_results_merger.cpp:518), which returns _highWaterMark as-is.

Notably RouterStagePipeline's constructor (src/mongo/s/query/exec/router_stage_pipeline.cpp:33-37) does seed the ARM with the correct value – expCtx->getInitialPostBatchResumeToken(), i.e. the verbatim resume token, set at src/mongo/db/pipeline/document_source_change_stream_transform.cpp:73. The V2 stage's initialize() runs later, on the first doGetNext(), and overwrites that good value with the degraded high-water-mark token.

The shard side is not at fault: PlanExecutorPipeline::_performChangeStreamsAccounting only synthesizes a high water mark when it is strictly greater than the current PBRT timestamp, and its initial PBRT is the verbatim resume token.

Why this is new in v9

Two commits, both ancestors of the 9.0 branch:

• 4cb44190ad1d95e6f12079c434a92ceff8ccd260 – SERVER-121636 (#49626), 2026-03-17. This is the fix that closed SERVER-121636, and it is what introduces the current form of the bug. Its diff on this file is exactly:

  -        _mergeCursors->setInitialHighWaterMark(_initializationResumeToken.toBSON());
  +        setHighWaterMark(_initializationResumeToken.getClusterTime());
  

  Before that commit the ARM was seeded with the actual resume token. The same commit also collapsed the setInitialHighWaterMark/setHighWaterMark pair into a single setter which, per its own commit message, "does not enforce any semantics" – removing the forward-only check that would otherwise have caught the downgrade.

• 0f7210ba4f4 – SERVER-52253, enabling featureFlagChangeStreamPreciseShardTargeting at FCV 9.0, which makes change streams on mongos use the V2 reader. This is what makes the buggy path reachable in production.

v8 has no ChangeStreamHandleTopologyChangeV2 stage at all, which is consistent with the reporter not seeing this on v8.

Suggested fix

Preserve the client's token rather than degrading it, at line 87:

_mergeCursors->setHighWaterMark(_initializationResumeToken.toBSON());

Nothing downstream needs the degraded form: getTimestampFromCurrentHighWaterMark() (line 259) parses only clusterTime, and openCursorsOnDataShards()'s initial-request check compares atClusterTime against _initializationResumeToken.getClusterTime().

Hardening follow-ups

• Make AsyncResultsMerger::setHighWaterMark() (src/mongo/s/query/exec/async_results_merger.cpp:550) enforce monotonicity via the existing _ensureHighWaterMarkIsMonotonicallyIncreasing() instead of accepting any value. That is the invariant SERVER-121636's fix deliberately deleted, and it would have caught this.
• Promote the dassert at src/mongo/s/query/exec/async_results_merger.cpp:530 to a real check so backwards movement is not silent in release builds.
• Audit change_stream_handle_topology_change_v2_stage.cpp:1542, setHighWaterMark(*_segmentStartTimestamp) in the segment-end path. It has the same shape – a bare Timestamp re-wrapped as a high-water-mark token – and can move the watermark backwards past an already-emitted event token at the same clusterTime.

Acceptance criteria

• A resumed cluster-wide change stream on mongos never returns a PBRT that compares less than the client's resumeAfter/startAfter token, including when the first batch is empty.
• Regression test covering that case.