-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
None
-
DB Integration & Observability
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
Shard removal is not consumer-aware: a change stream reader that has not consumed past the removal point loses its footing when the removed shard's nodes are shut down, and nothing guarantees it gets the chance to catch up first.
The V2 reader (9.0, SPM-1941 — the primary reader going forward) is placement-driven: cursors exist only on shards that own the watched data, and the drain retires them per-collection — placement changes emit control events (moveChunk with allCollectionChunksMigratedFromDonor, movePrimary, namespacePlacementChanged), and when the last chunk of a collection leaves a shard the reader closes that shard's cursor (closeCursorsOnDataShards). As a result:
- A caught-up reader is fully protected: a fully-drained shard holds no V2 cursors, so the removal commit and the node shutdown are invisible to it.
- A lagging reader — whose read position predates the drain and still needs the removed shard's history — is not protected. When the nodes are shut down (T2), strict mode fails the stream fatally (non-resumable ShardRemovedError), and ignoreRemovedShards mode skips the missing history (resumable RetryChangeStream, potential event loss).
The core gap: nothing ensures a lagging reader reaches the removal point before the nodes die. The window between the removal commit (T1) and the shutdown (T2) is controlled by the deployment layer, not the server (see CLOUDP-456492: Atlas shuts the removed shard's nodes down immediately after the removal commit) — so the reader must win a race it does not control, and the "safe" outcome (strict, no data loss) is not guaranteed.
Moreover, the removal commit invalidates the existing safe-removal runbook: the TSE script recommended for orchestrating shard removals with open change streams (the change_streams_and_remove_shard.js runbook) collects resume tokens for all open streams before issuing the final removeShard (each token post-dating the drain-empty point "tSafe"), and re-establishes the streams from those tokens after the removal interrupts them. Because those tokens pre-date the removal commit, the SERVER-90358 safeguard rejects them on 8.0 — the runbook's final step fails with ChangeStreamHistoryLost.
For completeness: V1 (the 8.0 reader) has the same class of problem with different symptoms — the removal is invisible to running streams (host-bound established cursors), the shutdown fails them with HostUnreachable, and re-opening from a token predating the removal commit fails with ChangeStreamHistoryLost (see SERVER-90358; verified in HELP-101679). Since V1 is expected to be removed in 10.0, the fix should target the V2 reader on 9.1+.
Proposal (sketch)
The governing invariant: a shard is safe to remove and shut down when no change stream reader's consumed point-in-time still requires the shard — i.e. every active reader has consumed past the placement-history entry in which the removed shard dropped out of the watched namespace's placement (the per-namespace drain completion point).
A note on why cursor presence is not a sufficient test: a reader lagging behind a placement-add event (its PIT predates the shard joining the namespace's placement) has no cursor on the shard yet, but its future includes events generated on the shard — cursor absence is ambiguous between "retired because passed" and "not yet opened because behind". The check must compare each reader's consumed PIT against the namespace's placement history, not count cursors.
Sketch:
- When draining completes, the per-namespace drain point is recorded (the placement history already records the placement change; the removal log records the removal itself).
- Track the consumed PIT (PBRT / last delivered event cluster time) of every active change stream reader against the per-namespace drain point. The tracking mechanism is an implementation detail to be designed — options include mongos-reported state (e.g. via the config.mongos heartbeat) or $currentOp-based inspection of per-shard cursors.
- commitShardRemoval rejects with a retryable error while any active reader's PIT is still behind the drain point; the deployment layer already polls the removal/drain status and retries, so the waiting lives in the existing control-plane loop. A force option allows bypassing (accepting potential event loss); abandoned (never-polled) cursors self-resolve via the cursor idle timeout.
- Re-anchor the resumability safeguard at the drain point instead of the removal commit: a resume point at or after the drain point is provably complete (the removed shard holds no user data after draining — no user events can exist between drain completion and the removal commit). This restores the TSE runbook described above: resume tokens collected after consuming past tSafe, but before the removal commit, must remain valid. This part benefits V1 (8.0) as well and can be considered for a backport independently.
With this, the node shutdown becomes safe whenever it happens: readers have provably passed the drain point, so strict mode never hits the fatal ShardRemovedError, no reader targets the dead host, and the deployment layer needs no coordination work.
Known limitations (to be documented)
- The server can only observe live readers. A stopped consumer (e.g. a down ASP processor) holds a checkpoint the server cannot see; if it does not restart before the nodes are shut down, its checkpoint predates the drain point and resuming still fails. The waiting phase is, however, a window of opportunity for the control plane to restart such consumers while the removed shard's oplog is still readable.
- Abandoned (never-polled) cursors delay finalization until their idle timeout kills them, or until force is used.
Related
- HELP-101679, HELP-101385, CLOUDP-456492