[SERVER-80427] Avoid change streams latency caused by lack of writes on a shard Created: 25/Aug/23 Updated: 12/Jan/24 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Katya Kamenieva | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 4 |
| Labels: | changestreams, query-product-urgency-2, query-product-value-1, sharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Query Execution
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Description |
|
In a sharded cluster, to guarantee total ordering of events, change stream checks for events on all shards. This constrains latency of the events in change stream by the largest latency between writes on any individual shard. Default for periodicNoopIntervalSecs is 10s, so in the worst case when one of the shards doesn't receive any writes, the latency will be around that. |
| Comments |
| Comment by Katya Kamenieva [ 25/Aug/23 ] |
|
One of the workarounds can be issuing artificial write traffic to all shards. E.g., create a job that inserts a small document with the required frequency, and remove them using TTL index. |
| Comment by Katya Kamenieva [ 25/Aug/23 ] |
|
It's possible to adjust periodicNoopIntervalSecs and set it as low as 1s, but it's still often not meeting the requirements of the practical use cases |