A single reshard operation can emit two reshardCollection change events (i.e., the completion noop oplog entry is written twice to the same shard). This is user-visible: a client consuming change streams receives the event twice for one reshard. It was observed in CI on the change-streams sharding FSM suites (3 failures in ~35 runs on the all-feature-flags variants, tracked in BF-47107) and reproduced deterministically with a diagnostic failpoint.

I spoke with nicola.cabiddu@mongodb.com, who is of the opinion that this is a break of expectations for change streams, and we should fix it at the retry layer. Some supporting evidence for that claim:

• _shardsvrNotifyShardingEventCommand::supportsRetryableWrite() returns true (src/mongo/db/global_catalog/ddl/shardsvr_notify_sharding_event_command.cpp:43), i.e. the command opts into retryable-write dedup on the receiving shard. But the coordinator always sends the notification with a fresh operation session txnNumber on retry (_getNewSession -> calculateNextSession, resharding_coordinator.inl:241-243, src/mongo/db/s/primary_only_service_helpers/operation_session_tracker.cpp:28-38). By construction the dedup key (session, txnNumber) can never match, so the dedup this layer advertises can never fire.
• Change-streams documentation and tests treat reshard as exactly-once (one reshardCollection event per operation); the FSM verifier models exactly one event per command.
• Duplicate events are visible to application clients and break exactly-once consumers (idempotent processing at the stream layer).

Included below are some details from my AI-based investigation:

Mechanism

ReshardingCoordinator::_commitAndFinishReshardOperation (resharding_coordinator.inl:634-661) runs the completion notification (BeforeWriteOnCatalog mode) and the decision-persist update config.reshardingOperations in one WithAutomaticRetry scope. When the config write is interrupted (e.g. an FCV flip in the FSM, or any transient), the retry re-sends the notification with the fresh txnNumber, the donor's retryable-write dedup misses, and the noop is written a second time. Note the emission-site comment says "multiple copies of this event notification are acceptable" - we ask the owning team to confirm whether that is still the intended contract; if duplicates are not intended (our position), the retryable-write opt-in and the exactly-once expectation in change streams mean this is a bug in the emission path.

Evidence from/related to the BF

• evergreen history: ~0/~600 runs Oct 1 - Nov 2, 3 failures Nov 3-4 (~8.5%) on change_streams_sharding_fsm_bg_mutator_embedded_watch_db (x2) and change_streams_sharding_fsm_embedded_watch_collection (x1 - no background mutator, so the FCV flip is a trigger, not the cause).
• Raw shard oplog of the failing run: one reshard op (reshardUUID 8dba47a6-c755-4c8a-90c6-1a4a0db63f21) wrote its noop twice on rs1 - ts (1793787145,28) and (1793787145,89) (same wall second, ~200 ms apart), with an interrupted update config.reshardingOperations between them and "Authoritative collection metadata recovery" immediately after.
• Not a new mechanism: identical retry/_getNewSession path exists in the 2026-10-08 tree (git log b721644b105c..HEAD over the emission files is empty; structure dates to SERVER-106740 / 2025-07-23, emission to SERVER-74031 / 2023). The Nov-2/3 window (esp. SERVER-136621 concurrent reshards) plausibly raised how often an interrupt lands in the notification->persist window - exposure up, mechanism unchanged.

Suggested fixes (this layer)

1. Preferred - coordinator: reuse a stable OSI for the notification send, derived from _coordinatorDoc.getReshardingUUID() (fixed lsid + txnNumber per operation), in _generateCommitNotificationForChangeStreams (resharding_coordinator.inl:2118-2125) and the placement-change sibling (:2151-2158). The donor's existing retryable-write dedup then collapses re-sends. Verify OSI ordering/causality + session-record TTL on shards.
2. Alternative - write-path idempotency: skip insertNotificationOplogEntries in notifyChangeStreamsOnReshardCollectionComplete (src/mongo/db/repl/change_stream_oplog_notification.cpp:126-211) if a noop with the same o2.reshardUUID + nss already exists (under the exclusive oplog write lock).
3. Or - drop supportsRetryableWrite() from _shardsvrNotifyShardingEventCommand if dedup was never intended, making the duplicate documented rather than accidentally unguarded.

Repro

See the repro kit referenced from BF-47107 (deterministic: ~55 s; natural-forced variant ~10-14%):

• DUPNOOP=1 ./repro_bf47107/run_mini_repro.sh -> one reshard writes 2 noops.
• FAILCMD=1 ./repro_bf47107/run_mini_repro.sh -> interrupts the decision-persist write, exercising the real retry path. (The failpoints are local diagnostic only.)

Related

• BF-47107, SERVER-138724 (interim test-side tolerance; removed once this is fixed)