[COMPASS-7001] Investigate changes in PM-1941: Improved change stream shard-targeting and handling of cluster topology changes Created: 05/Jul/23  Updated: 07/Feb/24

Status: Needs Triage
Project: Compass
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Investigation Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Epic Link: COMPASS-7048
Sprint: Up for triaging

 Description   
Original Downstream Change Summary

New parameters added to $changeStream aggregation stage.

Description of Linked Ticket

Epic Summary

Summary

Change streams on sharded clusters should return correct events when shards are removed or added.

Motivation

This has been a user request and will allow us to offer change streams for more use cases while increasing client-side debugability. There are scenarios that may lead to incorrect events for change stream users on sharded clusters. 

For example: if a shard gets removed from a cluster before we are finished reading events from it and we resume a stream from a point before the shard was removed, we will silently skip every event that occurred on the removed shard between that time and the point where it was removed. This violates one of change streams’ fundamental guarantees (no skipped events).

In a serverless world, where we want to be able to add and remove shards at will, this will become an important issue to fix. 

Cast of Characters

  • Product Owner: Katya Kamenieva
  • Project Lead: Mindaugas Malinauskas
  • Program Manager: Ana Meza
  • Drivers Contact: TBD

Documentation

Scope Document
Technical Design Document
Product Description
Docs Update

Project Dashboard



 Comments   
Comment by Anna Henningsen [ 07/Feb/24 ]

We might want to wait for the outcome of the conversation in https://docs.google.com/document/d/1UqPXCCMJtsWFnNlcTKQE63_Ov5Chi4DQTLayT6Rbbpk/edit?disco=AAABF7QKKi4 to check whether this will actually still be relevant.

Comment by Anna Henningsen [ 05/Feb/24 ]
  • The new parameter, according to the scope doc is a boolean named ignoreRemovedShards
  • Only change for us that makes sense here would be to add it to the template in mongodb-constants
  • We probably need to decide whether we want to include this parameter for completeness or just don't care because it's only for specific use cases that DevTools probably doesn't target
Comment by PM Bot [ 08/Jan/24 ]

Fix Version updated for upstream PM-1941:
8.1 Targeted

Comment by PM Bot [ 19/Jul/23 ]

Fix Version updated for upstream PM-1941:

Generated at Wed Feb 07 22:45:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.