[SERVER-47600] Submitting orphan ranges during setFCV may fail if MetadataManager::_receivingChunks is not empty Created: 16/Apr/20  Updated: 29/Oct/23  Resolved: 20/Apr/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.4.0-rc2, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Gregory Noma Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

The generated tests can be downloaded here. Running jstestfuzz-d133-ent_b819-qa_a6ce-1586874860847-0.js and jstestfuzz-d133-ent_b819-qa_a6ce-1586874860847-1.js (in that order) together in the same resmoke invocation should cause a hang.

Sprint: Sharding 2020-05-04
Participants:
Linked BF Score: 50

 Description   

If MetadataManager::_receivingChunks is not empty (possibly due to a metadata refresh not occurring after a previous migration), the setFCV command can end up hitting a RangeOverlapConflict here. Thus the setFCV gets stuck which also causes subsequent migrations to pause indefinitely.



 Comments   
Comment by Githook User [ 21/Apr/20 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-47600 Clear _receivingChunks on setFCV after metadata refresh

(cherry picked from commit 5943a22ad1a7bc8c8bc0a855087a9992ef4a2524)
Branch: v4.4
https://github.com/mongodb/mongo/commit/1b18276d8c980dbe14e5f44f216d40b02904374e

Comment by Githook User [ 20/Apr/20 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-47600 Clear _receivingChunks on setFCV after metadata refresh
Branch: master
https://github.com/mongodb/mongo/commit/5943a22ad1a7bc8c8bc0a855087a9992ef4a2524

Comment by Esha Maharishi (Inactive) [ 16/Apr/20 ]

The problem:

If a recipient finishes its end of a migration successfully, but the migration fails to commit on the config server, the range  remains in the recipient's _receivingRanges. This is because the recipient only removes the range from _receivingRanges if it sees that it now owns that range. (The range would also be removed if the recipient re-received the range, because MigrationDestinationManager::_migrateDriver -> MigrationDestinationManager::_notePending -> CollectionShardingRuntime::beginReceive -> MetadataManager::beginReceive would emplace the same range in _receivingChunks and continue with that second migration. If the second migration succeeded, the range would be removed from the recipient's _receivingRanges when the recipient refreshed after the second migration's commit.)

If setFCV inserts a range deletion task for the range (or an overlapping range) while the range is stuck in _receivingRanges after a failed migration commit, processing the range deletion task will fail with RangeOverlapConflict. At that point, a second migration will block waiting for the range deletion task to be processed.

Potential solutions:

setFCV has a special property that the refresh a shard does as part of the setFCV's submitOrphanRangesForCleanup is guaranteed to be at or after the time the config server wrote that it was starting to upgrade its FCV. This is because the config server sends setFCV to all shards and gossips its configOpTime on all messages, and shards refresh with an afterOpTime of the latest configOpTime they know about.

Since any migration that had not committed by the time the setFCV began on the config server will not be able to commit afterwards*, the shard can safely assume that after it refreshes as part of setFCV, the set of ranges it owns will not change.

We can exploit this to ensure the range deletion task for setFCV is able to complete successfully even if there is a range stuck in _receivingRanges by:

*I remember why we had decided not to clear _receivingRanges on setFCV. It's because a migration commit could succeed if the FCV was upgraded and then downgraded.Then the recipient would officially own the chunk, but would have deleted the chunk's data. I think this is too unlikely a scenario to really worry about, though. People probably do not setFCV to upgrade and then setFCV to downgrade quickly enough for this to happen in practice.

Generated at Thu Feb 08 05:14:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.