[SERVER-67529] Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern Created: 24/Jun/22  Updated: 29/Oct/23  Resolved: 26/Jul/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 5.0.3, 6.0.0-rc11, 7.0.0-rc8
Fix Version/s: 7.1.0-rc0, 7.0.1, 6.0.10, 5.0.21

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Tommaso Tocci
Resolution: Fixed Votes: 0
Labels: shardingemea-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-71627 Refreshed cached collection route inf... Closed
Related
related to SERVER-79549 Audit and perhaps consolidate the mul... Backlog
is related to SERVER-38207 Cannot insert document with MaxKey sh... Backlog
is related to SERVER-57667 Improve processing speed for reshardi... Closed
Assigned Teams:
Sharding EMEA
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0, v6.0, v5.0
Sprint: Sharding EMEA 2022-08-22, Sharding EMEA 2022-09-05, Sharding EMEA 2022-11-14, Sharding EMEA 2022-11-28, Sharding EMEA 2022-12-12, Sharding EMEA 2022-12-26, Sharding EMEA 2023-01-09, Sharding EMEA 2023-07-24, Sharding EMEA 2023-08-07
Participants:
Story Points: 2

 Description   

The $_internalReshardingOwnershipMatch aggregation stage is used by resharding to efficiently determine whether or not a document belongs to a particular shard under the new shard key pattern. It uses ChunkManager::keyBelongsToShard() to make this determination. However, findIntersectingChunk() will return nullptr and cause keyBelongsToShard() to always return false for a document {newKey1: MaxKey, newKey2: MaxKey} and a new shard key pattern {newKey1: 1, newKey2: 1}. This leads to no shard receiving the document and it being entirely lost as part of the resharding operation.

Note that once the resharding operation has started, the participant shards won't allow inserting more documents or updating existing documents to have all MaxKey values for its fields under the new shard key pattern.

{
 	"nMatched" : 0,
 	"nUpserted" : 0,
 	"nModified" : 0,
 	"writeError" : {
 		"code" : 61,
 		"errmsg" : "Cannot target single shard using key { newKey: MaxKey } for namespace reshardingDb.system.resharding.0407a566-91d8-4152-aa4a-6014f161680f"
 	}
 }



 Comments   
Comment by Githook User [ 25/Aug/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-67529 Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern

(cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac)
(cherry picked from commit 5f9ead7b2f556432065fa115b80e6b1b9e9c3958)
(cherry picked from commit fadba154f60803de29752dd55218cbfeb8f969ae)
Branch: v5.0
https://github.com/mongodb/mongo/commit/824ab22fc777ecbfd7f408698d6064a1a6f544e6

Comment by Githook User [ 24/Aug/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-67529 Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern

(cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac)
(cherry picked from commit 5f9ead7b2f556432065fa115b80e6b1b9e9c3958)
Branch: v6.0
https://github.com/mongodb/mongo/commit/7867f5e7268722d17ade99a32e1c24331c5f783a

Comment by Githook User [ 18/Aug/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-67529 Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern

(cherry picked from commit b8e050ad94a6a584a15df5744c6ae59d216b42ac)
Branch: v7.0
https://github.com/mongodb/mongo/commit/5f9ead7b2f556432065fa115b80e6b1b9e9c3958

Comment by Githook User [ 26/Jul/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-67529 Resharding silently skips documents with all MaxKey values for their fields under the new shard key pattern
Branch: master
https://github.com/mongodb/mongo/commit/b8e050ad94a6a584a15df5744c6ae59d216b42ac

Comment by Marcos José Grillo Ramirez [ 11/Jan/23 ]

This ticket should wait until the changes related to SERVER-71627 are committed.

Comment by Marcos José Grillo Ramirez [ 25/Nov/22 ]

Passing this ticket to sergi.mateo-bellido@mongodb.com because I'll be OOO for the next sprint. Please have a look at the current POC.

Comment by Max Hirschhorn [ 24/Jun/22 ]

I'm assigning this ticket to Sharding EMEA because the bug fix likely has to do with the std::upper_bound logic in ChunkMap::_findIntersectingChunk(). Every shard key value should fall within some chunk range. We should investigate whether addressing the faulty logic in ChunkManager is sufficient for addressing SERVER-38207 or if other work is necessary to have all BSON types work appropriately for sharded collections.

Generated at Thu Feb 08 06:08:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.