[SERVER-47600] Submitting orphan ranges during setFCV may fail if MetadataManager::_receivingChunks is not empty Created: 16/Apr/20 Updated: 29/Oct/23 Resolved: 20/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 4.4.0-rc2, 4.7.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Gregory Noma | Assignee: | Gregory Noma |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v4.4
|
||||||||
| Steps To Reproduce: | The generated tests can be downloaded here. Running jstestfuzz-d133-ent_b819-qa_a6ce-1586874860847-0.js and jstestfuzz-d133-ent_b819-qa_a6ce-1586874860847-1.js (in that order) together in the same resmoke invocation should cause a hang. |
||||||||
| Sprint: | Sharding 2020-05-04 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 50 | ||||||||
| Description |
|
If MetadataManager::_receivingChunks is not empty (possibly due to a metadata refresh not occurring after a previous migration), the setFCV command can end up hitting a RangeOverlapConflict here. Thus the setFCV gets stuck which also causes subsequent migrations to pause indefinitely. |
| Comments |
| Comment by Githook User [ 21/Apr/20 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: (cherry picked from commit 5943a22ad1a7bc8c8bc0a855087a9992ef4a2524) |
| Comment by Githook User [ 20/Apr/20 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: |
| Comment by Esha Maharishi (Inactive) [ 16/Apr/20 ] |
|
The problem: If a recipient finishes its end of a migration successfully, but the migration fails to commit on the config server, the range remains in the recipient's _receivingRanges. This is because the recipient only removes the range from _receivingRanges if it sees that it now owns that range. (The range would also be removed if the recipient re-received the range, because MigrationDestinationManager::_migrateDriver -> MigrationDestinationManager::_notePending -> CollectionShardingRuntime::beginReceive -> MetadataManager::beginReceive would emplace the same range in _receivingChunks and continue with that second migration. If the second migration succeeded, the range would be removed from the recipient's _receivingRanges when the recipient refreshed after the second migration's commit.) If setFCV inserts a range deletion task for the range (or an overlapping range) while the range is stuck in _receivingRanges after a failed migration commit, processing the range deletion task will fail with RangeOverlapConflict. At that point, a second migration will block waiting for the range deletion task to be processed. Potential solutions: setFCV has a special property that the refresh a shard does as part of the setFCV's submitOrphanRangesForCleanup is guaranteed to be at or after the time the config server wrote that it was starting to upgrade its FCV. This is because the config server sends setFCV to all shards and gossips its configOpTime on all messages, and shards refresh with an afterOpTime of the latest configOpTime they know about. Since any migration that had not committed by the time the setFCV began on the config server will not be able to commit afterwards*, the shard can safely assume that after it refreshes as part of setFCV, the set of ranges it owns will not change. We can exploit this to ensure the range deletion task for setFCV is able to complete successfully even if there is a range stuck in _receivingRanges by:
*I remember why we had decided not to clear _receivingRanges on setFCV. It's because a migration commit could succeed if the FCV was upgraded and then downgraded.Then the recipient would officially own the chunk, but would have deleted the chunk's data. I think this is too unlikely a scenario to really worry about, though. People probably do not setFCV to upgrade and then setFCV to downgrade quickly enough for this to happen in practice. |