[SERVER-73170] Ensure new shard key pattern is used for extracting document key in oplog entries for change streams to consume post refineCollectionShardKey Created: 21/Jan/23  Updated: 26/Oct/23

Status: Backlog
Project: Core Server
Component/s: Change streams, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Max Hirschhorn Assignee: Backlog - Catalog and Routing
Resolution: Unresolved Votes: 0
Labels: oldshardingemea
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Catalog and Routing
Participants:

 Description   

The refineCollectionShardKey command does not cause each of the shards to refresh. This means until the shard happens to refresh its sharding metadata from the config server for another reason, it will continue to use the pre-refined shard key pattern. Oplog entries generated by the replica set shard primary won't include the newly-added suffix fields to the shard key pattern until the shard has refreshed. The documentKey field reported in change events in based on the fields extracted and recorded in the oplog entry and therefore can lead to a sequence where change events after the refineCollectionShardKey change event won't include the newly-added suffix fields to the shard key pattern.



 Comments   
Comment by Max Hirschhorn [ 21/Jan/23 ]

Kal had also brought up a where because the vector clock is not checkpointed on disk, it is possible for a replica set shard secondary which steps up to be primary to refresh from a lagged config server secondary and not see the effects of the refineCollectionShardKey either. This can mean oplog entries generated by the shard may start to include the newly-added suffix fields to the shard key pattern and then stop and then start again.

Generated at Thu Feb 08 06:23:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.