Problem

On a sharded collection, a change stream with fullDocument: "updateLookup" performs the post-image read as a routed remote operation (SERVER-31665 semantics): it obeys the client's read preference (majority readConcern with afterClusterTime), and node selection is re-rolled per event among the read-preference-eligible secondaries of the owning shard.

This read has no deadline:

• maxTimeMS on a getMore, whether set by the client (maxAwaitTimeMS) or by the router (the 1s cap it puts on shard getMores), bounds only the await-for-new-data phase, not the getMore's processing (oplog scanning and per-event enrichment, including the lookup). The hung getMores observed in production carried maxTimeMS: 1000 and ran for 84 minutes.
• internal node-to-node connections have no socket timeout.

If the selected member is unresponsive, i.e. up and accepting connections but not processing (e.g. a storage-level stall), the lookup request is delivered onto an established connection and never answered. No error is ever produced, so no retry or fail-over logic applies: the getMore blocks inside the lookup indefinitely. The change stream emits no further events and raises no error; from the consumer's perspective the stream silently stops.

Plain replica sets are not affected: on non-shard-server mongods the lookup is always performed locally (ShardTargetingPolicy::kNotAllowed). Within a sharded cluster, however, the routed path applies to all collections, sharded and unsharded alike, since it is selected by the process interface, not by the collection's sharding status.

Blast radius

Node selection re-rolls per event, and a shard-side getMore processes a batch bounded only by 16 MB of events (no batchSize forwarded, SERVER-102454). A single unresponsive member in the owning shard's replica set is therefore hit by every actively-delivering stream within seconds (P ≈ 1−(1−1/N)^n per getMore), and the change stream merger gates each stream on its own cursors advancing, so the frozen cursors block the streams entirely, for as long as the impaired node remains up.

Observed in production (HELP-101466, 8.0.34): one impaired secondary out of 17 froze all 32 change streams on the cluster for the full 84-minute duration of the node's impairment. All hung getMores failed simultaneously only when the node died. The replica set monitor marked the node DOWN about 10 minutes into the incident, after every hang had begun, and in-flight reads are never retargeted.

Exposure by version

• 8.0: every updateLookup is routed. The impairment of any single read-preference-eligible secondary can freeze all streams on the owning shard.
• 9.0+/master (SPM-4535): locally-owned lookups are performed in-process; the common case no longer routes, greatly reducing the probability. Remaining routed lookups: cross-shard lookups (the balancer commits a migration between the event and its lookup; probability scales with consumer lag) and scatter lookups for documents missing shard-key fields (SERVER-105706). The hang capability remains for these cases.

Reproduction

The test attached to this ticket (jstests/sharding/query/change_streams/change_stream_update_lookup_unbounded_wait.js) deterministically reproduces the hang on the legacy (routed) path: a stream pinned to one secondary via read-preference tag; a reconfig moves the tag so subsequent lookups target a different secondary; that secondary is made unable to serve the read (replication paused, so afterClusterTime is unsatisfiable); the routed lookup then blocks indefinitely. No timeout fires, no error is raised, the stream makes no progress, until an external intervention changes the target node's state, at which point the stream resumes.

Requested behavior

A change stream must not be able to hang indefinitely on a single lookup. A lookup that cannot complete within a bounded time must surface to the consumer as a resumable failure, allowing the stream to retry with re-targeting rather than silently freezing.

Possible direction (proposal, not agreed)

Bound remote updateLookups via a server parameter or query knob, enforced on the awaiting side (a deadline on the sender's wait; a maxTimeMS attached to the routed command alone only bounds targets that are slow-but-alive, since a stalled target never reaches a deadline check). On expiry, retry K times (reusing existing retry primitives) and only then fail with a dedicated error code/message that change stream consumers (and SPM-4784) can catch and surface as a special error event.

Links

• SERVER-31665: origin of the routed lookup semantics (correctness without a liveness bound)
• SERVER-105706, SERVER-68366: documentKey completeness / updateLookup broadcast
• SERVER-102454: batchSize not forwarded to shards
• SERVER-102900, SERVER-130542: internalOperationResponseMaxMS (yield-time checks; they do not bound a lookup already waiting on the network)
• SPM-4535: optimized in-process updateLookup
• HELP-101466: production incident (evidence chain in the ticket)