[SERVER-69220] refineCollectionShardKey permits toggling current shard key fields between range-based and hashed, leading to data inconsistency Created: 28/Aug/22 Updated: 29/Oct/23 Resolved: 07/Sep/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 4.4.0, 5.0.0, 6.0.0, 6.1.0-rc1 |
| Fix Version/s: | 4.4.17, 5.0.13, 6.0.2, 6.1.0-rc2, 6.2.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Enrico Golfieri |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | shardingemea-qw | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v6.1, v6.0, v5.0, v4.4
|
||||||||||||||||
| Sprint: | Sharding EMEA 2022-09-19 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Story Points: | 7.5 | ||||||||||||||||
| Description |
|
Refining a collection's shard key pattern modifies the sharding metadata stored on the config server without triggering any physical data movement. Let's say the collection's shard key pattern is refined from {x: 1} to {x: 1, y: 1}. The observation is when the chunk bounds were expressed only in terms of 'x' values, for a particular value of 'x', all the possible values of 'y' lived on that one shard. So padding the chunk bounds in the sharding metadata when the collection's shard key pattern is refined expresses that equivalency and allows the balancer + auto-splitter to later split the ranges into smaller ones and migrate them. However, the ShardKeyPattern::isExtendedBy() function called by the _configsvrRefineCollectionShardKey command only checks whether the field names in the new shard key pattern have the current shard key pattern as a prefix. In particular, it doesn't validate that the value for each of the field names (i.e. 1 or "hashed") is the same between the two shard key patterns. This unintentionally permits the collection's shard key pattern {x: 1} to be "refined" to {x: "hashed"} when there's a supporting index. However, the existing data is distributed using {x: 1} and not {x: "hashed"} so some of the existing data will become inaccessible. The existing data is still present on all of the shards and can be queried when using read concern level "available" but likely unowned by the shard and therefore cannot be updated.
We should change ShardKeyPattern::isExtendedBy() is perform this additional validation to prevent users from getting into a bad state. |
| Comments |
| Comment by Githook User [ 12/Sep/22 ] |
|
Author: {'name': 'Enrico Golfieri', 'email': 'enrico.golfieri@mongodb.com', 'username': 'enricogolfieri'}Message: |
| Comment by Githook User [ 08/Sep/22 ] |
|
Author: {'name': 'Enrico Golfieri', 'email': 'enrico.golfieri@mongodb.com', 'username': 'enricogolfieri'}Message: |
| Comment by Githook User [ 08/Sep/22 ] |
|
Author: {'name': 'Enrico Golfieri', 'email': 'enrico.golfieri@mongodb.com', 'username': 'enricogolfieri'}Message: |
| Comment by Githook User [ 07/Sep/22 ] |
|
Author: {'name': 'Enrico Golfieri', 'email': 'enrico.golfieri@mongodb.com', 'username': 'enricogolfieri'}Message: |