[SERVER-61483] Resharding coordinator fails to recover abort decision on step-up, attempts to commit operation as success, leading to data inconsistency Created: 15/Nov/21  Updated: 29/Oct/23  Resolved: 17/Nov/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 5.0.0, 5.1.0
Fix Version/s: 5.2.0, 5.0.5, 5.1.1

Type: Bug Priority: Critical - P2
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: sharding-nyc-subteam1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-61482 Updates to config.reshardingOperation... Closed
is related to SERVER-61473 Resharding coordinator calls Reshardi... Closed
is related to SERVER-50937 Make resharding coordinator support r... Closed
is related to SERVER-52770 Add abortReshardCollection command fo... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.1, v5.0
Sprint: Sharding 2021-11-29
Participants:
Story Points: 2

 Description   

The ReshardingCoordinator relies on an exception being thrown and its .onError() handler being called to trigger its _shardsvrAbortReshardCollection flow. However, the ReshardingCoordinator fails to read the current state of the coordinator document to trigger the _shardsvrAbortReshardCollection flow when an earlier config server primary had already decided the resharding operation must abort. The lack of the .onError() handler being called leads the ReshardingCoordinator to attempt to commit the resharding operation anyway. This is severely problematic because the resulting collection will be incomplete and inconsistent (i.e. lost writes).

  • Shards which had already received the _shardsvrAbortReshardCollection command from the earlier config server primary's resharding coordinator may have dropped the temporary resharding collection already. These shards effectively ignore the _shardsvrCommitReshardCollection command.
  • Other shards which erroneously receive the _shardsvrCommitReshardCollection command will rename the temporary resharding collection over the source collection.
    • Even shards which voted to abort to abort resharding operation (e.g. unrecoverable error during collection cloning or oplog application) can still rename the temporary resharding collection over the source collection.
    • However shards which aren't in the "strict-consistency" state (recipient role) and aren't in the "blocking-writes" state (donor role) will reject the _shardsvrCommitReshardCollection command. The ReshardCollectionInProgress error response returned to the resharding coordinator will lead the config server primary to fassert(). While the fassert(5277000) is an indicator of this issue occurring, it isn't guaranteed that any shards will still be in a state to detect the resharding coordinator having delivered different decisions to different shards.

{"t":{"$date":"2021-11-14T16:37:49.291+00:00"},"s":"E",  "c":"ASSERT",   "id":4457000, "ctx":"conn84","msg":"Tripwire assertion","attr":{"error":{"code":338,"codeName":"ReshardCollectionInProgress","errmsg":"Attempted to commit the resharding operation in an incorrect state"},"location":"{fileName:\"src/mongo/db/s/resharding/resharding_recipient_service.cpp\", line:918, functionName:\"operator()\"}"}}
{"t":{"$date":"2021-11-14T16:38:00.557+00:00"},"s":"F",  "c":"RESHARD",  "id":5277000, "ctx":"ReshardingCoordinatorService-1","msg":"Unrecoverable error past the point resharding was guaranteed to succeed","attr":{"error":"ReshardCollectionInProgress: Failed command { _shardsvrCommitReshardCollection: \"reshardingDb.coll\", reshardingUUID: UUID(\"4755e8fb-35ab-4306-b832-c3a81b44b8d1\"), writeConcern: { w: \"majority\" }, $audit: { $impersonatedUsers: [ { user: \"__system\", db: \"local\" } ], $impersonatedRoles: [] } } for database 'admin' on shard 'shard1-recipient0' :: caused by :: Attempted to commit the resharding operation in an incorrect state"}}


Thank you to chuck.zhang for discovering this issue while working on the automation restore procedure (which has the config server being started up in the aborting state for the resharding operation).



 Comments   
Comment by Max Hirschhorn [ 17/Nov/21 ]

The 5.0 backport was split into two commits to enable the changes from 963c540 as part of SERVER-61482 being committed in between. My patch runs on the 5.0 branch revealed the resharding_coordinator_recovers_abort_decision.js test could sometimes experience stalls in replication on the config server as well. I decided to put the fix in from SERVER-61482 after committing the ReshardingTest fixture change dependency to avoid introducing any redness to the Evergreen build.

Comment by Githook User [ 17/Nov/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-61483 Fix resharding coordinator to recover its abort decision.

(cherry picked from commit d9fcd9f124ece9ab0b3a3c46cb6d7052b7282dd2)
Branch: v5.0
https://github.com/mongodb/mongo/commit/756c586e7fc94dd923bfcb20630f88c33f243795

Comment by Githook User [ 17/Nov/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-61483 Retry once in ReshardingTest when disabling failpoints.

(cherry picked from commit d9fcd9f124ece9ab0b3a3c46cb6d7052b7282dd2)
Branch: v5.0
https://github.com/mongodb/mongo/commit/4c57f6bca3334bee1118a695d871db5346c75ff5

Comment by Githook User [ 16/Nov/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-61483 Fix resharding coordinator to recover its abort decision.

(cherry picked from commit d9fcd9f124ece9ab0b3a3c46cb6d7052b7282dd2)
Branch: v5.1
https://github.com/mongodb/mongo/commit/201967a1696da1340f5dd7e328fe1229667e8e36

Comment by Githook User [ 15/Nov/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-61483 Fix resharding coordinator to recover its abort decision.
Branch: master
https://github.com/mongodb/mongo/commit/d9fcd9f124ece9ab0b3a3c46cb6d7052b7282dd2

Generated at Thu Feb 08 05:52:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.