[SERVER-67529] Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern Created: 24/Jun/22 Updated: 29/Oct/23 Resolved: 26/Jul/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 5.0.3, 6.0.0-rc11, 7.0.0-rc8 |
| Fix Version/s: | 7.1.0-rc0, 7.0.1, 6.0.10, 5.0.21 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Tommaso Tocci |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | shardingemea-qw | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Sharding EMEA
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Backport Requested: |
v7.0, v6.0, v5.0
|
||||||||||||||||||||||||||||
| Sprint: | Sharding EMEA 2022-08-22, Sharding EMEA 2022-09-05, Sharding EMEA 2022-11-14, Sharding EMEA 2022-11-28, Sharding EMEA 2022-12-12, Sharding EMEA 2022-12-26, Sharding EMEA 2023-01-09, Sharding EMEA 2023-07-24, Sharding EMEA 2023-08-07 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||||||||||
| Description |
|
The $_internalReshardingOwnershipMatch aggregation stage is used by resharding to efficiently determine whether or not a document belongs to a particular shard under the new shard key pattern. It uses ChunkManager::keyBelongsToShard() to make this determination. However, findIntersectingChunk() will return nullptr and cause keyBelongsToShard() to always return false for a document {newKey1: MaxKey, newKey2: MaxKey} and a new shard key pattern {newKey1: 1, newKey2: 1}. This leads to no shard receiving the document and it being entirely lost as part of the resharding operation. Note that once the resharding operation has started, the participant shards won't allow inserting more documents or updating existing documents to have all MaxKey values for its fields under the new shard key pattern.
|
| Comments |
| Comment by Githook User [ 25/Aug/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: (cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac) |
| Comment by Githook User [ 24/Aug/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: (cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac) |
| Comment by Githook User [ 18/Aug/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: (cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac) |
| Comment by Githook User [ 26/Jul/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Marcos José Grillo Ramirez [ 11/Jan/23 ] |
|
This ticket should wait until the changes related to SERVER-71627 are committed. |
| Comment by Marcos José Grillo Ramirez [ 25/Nov/22 ] |
|
Passing this ticket to sergi.mateo-bellido@mongodb.com because I'll be OOO for the next sprint. Please have a look at the current POC. |
| Comment by Max Hirschhorn [ 24/Jun/22 ] |
|
I'm assigning this ticket to Sharding EMEA because the bug fix likely has to do with the std::upper_bound logic in ChunkMap::_findIntersectingChunk(). Every shard key value should fall within some chunk range. We should investigate whether addressing the faulty logic in ChunkManager is sufficient for addressing SERVER-38207 or if other work is necessary to have all BSON types work appropriately for sharded collections. |