[SERVER-46386] Refining a shard key may lead to an orphan range never being cleaned up Created: 25/Feb/20  Updated: 29/Oct/23  Resolved: 13/Mar/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Matthew Saltz (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File server42192.patch    
Issue Links:
Backports
Depends
is depended on by SERVER-48198 Migration recovery may recover incorr... Closed
Related
is related to SERVER-46370 Correctly maintain receiving chunks l... Closed
is related to SERVER-42192 Write a concurrency workload to test ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

Apply server42192.patch to allow the moveChunk command to be automatically retried in the presence of failovers and run the random_moveChunk_refine_collection_shard_key.js FSM workload. The --repeat is necessary because while this concurrency test reproduces the issue often, it doesn't happen every time.

python buildscripts/resmoke.py --suite=concurrency_sharded_multi_stmt_txn_terminate_primary jstests/concurrency/fsm_workloads/random_moveChunk_refine_collection_shard_key.js --repeat=10

Sprint: Sharding 2020-03-09, Sharding 2020-03-23
Participants:

 Description   

The config.rangeDeletions collection stores a document of the following form to track range deletion tasks needing to be performed:

{
    "_id" : UUID("78447b8a-84d6-4555-a97d-b4bc2d708e29"),
    "nss" : "test4_fsmdb0.fsmcoll0_29",
    "collectionUuid" : UUID("1eafdb97-2114-4b72-a6ce-34e96aa3df1a"),
    "donorShardId" : "shard-rs1",
    "range" : {
        "min" : {
            "a" : 400
        },
        "max" : {
            "a" : { "$maxKey" : 1 }
        }
    },
    "whenToClean" : "delayed"
}

The "range" field of this document is left unmodified after a user has successfully run the refineCollectionShardKey command. This means if the following sequence of events occurs, then a "RangeOverlapConflict: Requested deletion range overlaps a live shard chunk" error will prevent the range deleter from ever deleting the range of orphan documents.

  1. The test4_fsmdb0.fsmcoll0_29 collection is sharded on {a: 1} and has chunks
    • {a: MinKey} -> {a: 100}
    • {a: 100} -> {a: 200}
    • {a: 200} -> {a: 300}
    • {a: 300} -> {a: 400}
    • {a: 400} -> {a: MaxKey}
  2. A migration begins for chunk {a: 400} -> {a: MaxKey} from shard-rs1 to shard-rs0.
  3. Migration completes successfully but the primary of shard-rs1 steps down before the range deletion task completes.
  4. User runs the refineCollectionShardKey and changes the shard key of the test4_fsmdb0.fsmcoll0_29 collection to {a: 1, b: 1}. The chunks are therefore augmented to be
    • {a: MinKey, b: MinKey} -> {a: 100, b: MinKey}
    • {a: 100, b: MinKey} -> {a: 200, b: MinKey}
    • {a: 200, b: MinKey} -> {a: 300, b: MinKey}
    • {a: 300, b: MinKey} -> {a: 400, b: MinKey}
    • {a: 400, b: MinKey} -> {a: MaxKey, b: MaxKey}
  5. The newly elected primary of shard-rs1 attempts to schedule the range deletion task still present in its config.rangeDeletions collection. Since the {a: 300, b: MinKey} -> {a: 400, b: MinKey} range is considered to partially overlap with the range {a: 400} -> {a: MaxKey} because {a: 400} < {a: 400, b: MinKey}, if shard-rs1 happens to own that range then it'll refuse to perform the range deletion for {a: 400, b: MinKey} -> {a: MaxKey, b: MaxKey}.


 Comments   
Comment by Githook User [ 30/Mar/20 ]

Author:

{'name': 'Matthew Saltz', 'email': 'matthew.saltz@mongodb.com', 'username': 'saltzm'}

Message: SERVER-46386 Change MetadataManager::_findNewestOverlappingMetadata to work with refined shard keys

(cherry picked from commit 46b4b3d8f03e3dfc3ca2ae344bf7848732e986c3)
Branch: v4.4
https://github.com/mongodb/mongo/commit/810b50de5ae4a21d91aaf0f644dd4be0e60ba592

Comment by Githook User [ 13/Mar/20 ]

Author:

{'username': 'saltzm', 'name': 'Matthew Saltz', 'email': 'matthew.saltz@mongodb.com'}

Message: SERVER-46386 Change MetadataManager::_findNewestOverlappingMetadata to work with refined shard keys
Branch: master
https://github.com/mongodb/mongo/commit/46b4b3d8f03e3dfc3ca2ae344bf7848732e986c3

Comment by Max Hirschhorn [ 25/Feb/20 ]

It is unknown to me if there may be other situations where ChunkManager::rangeOverlapsShard() or RoutingTableHistory::overlappingRanges() are called with a ChunkRange that has a shape from prior to the refinement of the shard key. I'd imagine those to be similarly problematic if there are any such places.

Generated at Thu Feb 08 05:11:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.