[SERVER-44039] Improve $changeStream handling of cluster topology changes Created: 16/Oct/19  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Bernard Gorman Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: change-streams-improvements, qexec-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-33090 $changeStream can fail if a shard was... Backlog
related to SERVER-48386 Adding a new shard while awaiting a f... Backlog
is related to SERVER-42723 New shard with new database can be ig... Closed
Assigned Teams:
Query Execution
Sprint: Query 2020-03-23, Query 2020-04-06, Query 2020-04-20
Participants:

 Description   

In SERVER-42723, we introduced an improved mechanism for change streams to detect the addition of a new shard to the cluster. This was motivated by the fact that the existing migrateChunkToNewShard mechanism was not sufficient to account for all potential cases. At present, the migrateChunkToNewShard oplog notification still exists in master for backward-compatibility and upgrade purposes, but we may wish to remove this during the 4.5 development cycle.

Alternatively, the two approaches could become complementary. At present, we must always open change stream cursors on all shards, and so the approach implemented by SERVER-42723 is always appropriate. However, in the future we may be able to target single-collection or single-DB streams to a subset of shards based on the historical state of the routing table at the moment from which the stream is resuming. In this scenario, the migrateChunkToNewShard approach is appropriate and the new SERVER-42723 mechanism would only be needed in the case of whole-cluster streams, or when a stream is opened on a database which does not yet exist.


Generated at Thu Feb 08 05:04:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.