[SERVER-60775] PrimaryOnlyService won't wait for prior Instance on step up if InstanceID had completed on an interceding primary already (ABA problem) Created: 18/Oct/21  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: 5.0.0, 5.1.0-rc0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Backlog - Service Architecture
Resolution: Unresolved Votes: 0
Labels: sa-remove-fv-backlog-22
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-57686 We need test coverage that runs resha... Closed
Related
is related to SERVER-54460 Resharding may delete the state docum... Closed
Assigned Teams:
Service Arch
Operating System: ALL
Participants:
Story Points: 1

 Description   

PrimaryOnlyService::onStepUp() has logic to wait for any Instances constructed in a previous term have finished executing before constructing any new Instances in a higher term.

// This ensures that all instances from previous term have joined.
for (auto& instance : savedInstances) {
    instance.second.waitForCompletion();
}

The logic in PrimaryOnlyService::onStepUp() only applies to Instances which are still tracked in PrimaryOnlyService::_activeInstances. When the state document for the Instance is removed, the Instance is also removed from PrimaryOnlyService::_activeInstances. However, PrimaryOnlyServiceOpObserver::onDelete() also run on secondaries as part of oplog application.

This leads to a situation where an Instance can be constructed in term 7 despite an (untracked) Instance with a different ID from term 5 not having its future returned by run() become ready. I suspect the solution here is to have PrimaryOnlyServiceOpObserver check whether the current node is primary when doing the delete/drop before removing the ActiveInstance from the map.

 

Acceptance criteria: Investigate the root cause and propose possible solutions for triage. 



 Comments   
Comment by Max Hirschhorn [ 18/Oct/21 ]

This issue can trigger to an invariant failure when running multiple resharding operations in sequence because DonorStateMachine and RecipientStateMachine assume they have exclusive access to the ReshardingMetrics.

[j0:s1:n0] | 2021-10-13T20:46:56.388+00:00 I  RESHARD  5279506 [ReshardingRecipientService-0] "Transitioned resharding recipient state","attr":{"newState":"cloning","oldState":"creating-collection","namespace":"test0_fsmdb0.fsmcoll0","collectionUUID":{"uuid":{"$uuid":"2d7cd853-4ab9-46c9-9672-910c3696c197"}},"reshardingUUID":{"uuid":{"$uuid":"c1206c8f-2c14-41cb-aafc-f23e664e0495"}}}
...
[j0:s1:n0] | 2021-10-13T20:47:00.995+00:00 I  REPL     21358   [ReplCoord-7] "Replica set state transition","attr":{"newState":"SECONDARY","oldState":"PRIMARY"}
...
[j0:s1:n2] | 2021-10-13T20:47:04.201+00:00 I  RESHARD  5279506 [ReshardingRecipientService-3] "Transitioned resharding recipient state","attr":{"newState":"done","oldState":"strict-consistency","namespace":"test0_fsmdb0.fsmcoll0","collectionUUID":{"uuid":{"$uuid":"2d7cd853-4ab9-46c9-9672-910c3696c197"}},"reshardingUUID":{"uuid":{"$uuid":"c1206c8f-2c14-41cb-aafc-f23e664e0495"}}}
...
[j0:s1:n0] | 2021-10-13T20:47:19.858+00:00 F  ASSERT   23081   [ReshardingRecipientService-2] "Invariant failure","attr":{"expr":"!_currentOp->recipientState","msg":"Another operation is in progress","file":"src/mongo/db/s/resharding/resharding_metrics.cpp","line":472}
[j0:s1:n0] | 2021-10-13T20:47:19.858+00:00 F  ASSERT   23082   [ReshardingRecipientService-2] "\n\n***aborting after invariant() failure\n\n"
[j0:s1:n0] | 2021-10-13T20:47:19.858+00:00 F  CONTROL  4757800 [ReshardingRecipientService-2] "Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}

https://evergreen.mongodb.com/lobster/build/61823411e61632ffd5de8754e7e4799b/test/61674598be07c441d96714a7/#bookmarks=0%2C20262%2C23125%2C24406%2C33374%2C33375%2C33376%2C321794&f~=000~%5C%5Bj0%3As1%3An0%5C%5D&f~=100~%5C%5Bj0%3As1.%2A%22%28Replica%20set%20state%20transition%7CTransitioned%20resharding%20recipient%20state%29%22&shareLine=33374

Generated at Thu Feb 08 05:50:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.