[SERVER-73170] Ensure new shard key pattern is used for extracting document key in oplog entries for change streams to consume post refineCollectionShardKey Created: 21/Jan/23 Updated: 26/Oct/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Change streams, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Max Hirschhorn | Assignee: | Backlog - Catalog and Routing |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | oldshardingemea | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Catalog and Routing
|
| Participants: |
| Description |
|
The refineCollectionShardKey command does not cause each of the shards to refresh. This means until the shard happens to refresh its sharding metadata from the config server for another reason, it will continue to use the pre-refined shard key pattern. Oplog entries generated by the replica set shard primary won't include the newly-added suffix fields to the shard key pattern until the shard has refreshed. The documentKey field reported in change events in based on the fields extracted and recorded in the oplog entry and therefore can lead to a sequence where change events after the refineCollectionShardKey change event won't include the newly-added suffix fields to the shard key pattern. |
| Comments |
| Comment by Max Hirschhorn [ 21/Jan/23 ] |
|
Kal had also brought up a where because the vector clock is not checkpointed on disk, it is possible for a replica set shard secondary which steps up to be primary to refresh from a lagged config server secondary and not see the effects of the refineCollectionShardKey either. This can mean oplog entries generated by the shard may start to include the newly-added suffix fields to the shard key pattern and then stop and then start again. |