[SERVER-54231] Resharding can leave behind local collection on former primary shard that doesn't own any chunks Created: 03/Feb/21 Updated: 29/Oct/23 Resolved: 07/Oct/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 5.0.2 |
| Fix Version/s: | 5.2.0, 5.0.4, 5.1.0-rc1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tommaso Tocci | Assignee: | Pierlauro Sciarelli |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | PM-234-M3, PM-234-T-lifecycle | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v5.1, v5.0
|
||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | Shuffle the allShardIds vector here in selectShardForNewDatabase() before to cycle on it. std::random_shuffle(allShardIds.begin(), allShardIds.end()); |
||||||||||||||||||||||||||||||||||||||||
| Sprint: | Sharding EMEA 2021-09-20, Sharding EMEA 2021-10-04, Sharding EMEA 2021-10-18 | ||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||||||||||||||||||||||
| Description |
Original summaryresharding_allowMigrations.js fails if primary shard of 'reshardingDb' is not the first shard Original descriptionI've discovered this while working on More specifically I've discovered that this happen only if the primary shard is chosen among one of the donor shards that is not shard-0. This is an example of a failing run on evg. |
| Comments |
| Comment by Githook User [ 12/Oct/21 ] | ||||||||||||||||||||||||||||
|
Author: {'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}Message: | ||||||||||||||||||||||||||||
| Comment by Githook User [ 11/Oct/21 ] | ||||||||||||||||||||||||||||
|
Author: {'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}Message: | ||||||||||||||||||||||||||||
| Comment by Githook User [ 07/Oct/21 ] | ||||||||||||||||||||||||||||
|
Author: {'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}Message: | ||||||||||||||||||||||||||||
| Comment by Pierlauro Sciarelli [ 28/Sep/21 ] | ||||||||||||||||||||||||||||
|
If we wanted to make a more narrowly scoped change to resharding, then I think we would want to add logic to the ReshardingCoordinator to broadcast a collection drop to all shards (not just those which own chunks for the sharded collection under the old key pattern) by the source UUID. This can now be done by calling the _shardsvrDropCollectionIfUUIDNotMatching implemented under I believe changing the movePrimary and moveChunk behavior would the preferred longer-term solution here. If we ever want to cleanup orphaned collection catalog entries, in addition to those changes we may also have to call the new command for cleaning up pre-existing garbage. I believe a reasonable plan would be:
| ||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 09/Sep/21 ] | ||||||||||||||||||||||||||||
To answer my question from before, the collection catalog entry doesn't get dropped part of donating the shard's last chunk for the collection. I believe changing the movePrimary and moveChunk behavior would the preferred longer-term solution here. If we wanted to make a more narrowly scoped change to resharding, then I think we would want to add logic to the ReshardingCoordinator to broadcast a collection drop to all shards (not just those which own chunks for the sharded collection under the old key pattern) by the source UUID. This could be done wholly after the participants have dropped/renamed the collection being resharded to avoid needing to think about what happens if the DonorStateMachine or RecipientStateMachine attempts to drop/rename the collection when the drop command comes in from the config server primary. | ||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 04/Feb/21 ] | ||||||||||||||||||||||||||||
kaloian.manassiev, no, the sequence of operations doesn't involve any concurrent DDL operations. The issue described in my earlier comment is how a former primary shard retains the collection catalog entry even when it no longer owns chunks for the sharded collection.
Tommaso had pointed out that the movePrimary command won't drop any sharded collections on the former primary shard even when it no longer owns chunks for them. What does chunk migration do on the donor shard when it is the last chunk for the sharded collection? Does the collection catalog entry get dropped on the donor shard? | ||||||||||||||||||||||||||||
| Comment by Kaloian Manassiev [ 04/Feb/21 ] | ||||||||||||||||||||||||||||
|
max.hirschhorn, the DDL project will make all DDL serialise with each other, so there should never be an attempt to run drop concurrently with a rename. Are you referring to the case where a command, which was "stuck" in a router somewhere comes much later? | ||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 03/Feb/21 ] | ||||||||||||||||||||||||||||
|
The DDL project is addressing this by having the drop collection command broadcasted to all shards rather than only shards which own a chunk for the collection. One thought would be to have the coordinator broadcast such a drop command. This drop collection command would need to use the collection UUID rather than its namespace string to avoid an ordering dependency with the collection rename on recipient shards to install the temporary resharding collection as the new sharded collection. | ||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 03/Feb/21 ] | ||||||||||||||||||||||||||||
|
I discussed this issue with Tommaso over Zoom. The resharding_allowMigrations.js test is failing because the donor1 shard still has the sharded collection on it despite the resharding operation having succeeded.
| ||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 03/Feb/21 ] | ||||||||||||||||||||||||||||
tommaso.tocci, I'm not sure what to make of this ticket which mentions resharding and is linked to I'd like to also mention that the ReshardingTest fixture runs the movePrimary command so the primary shard within the test is always the same. |