[SERVER-65469] Don't invariant in PrimaryOnlyService::lookupInstance during a lock-free read Created: 11/Apr/22  Updated: 12/Apr/22  Resolved: 12/Apr/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Brett Nawrocki Assignee: Brett Nawrocki
Resolution: Won't Fix Votes: 0
Labels: sharding-nyc-subteam1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-51650 Primary-Only Service's _rebuildCV sho... Closed
is related to SERVER-64372 Track and Report Parity Donor Fields ... Closed
Operating System: ALL
Sprint: Sharding NYC 2022-04-18
Participants:
Linked BF Score: 135
Story Points: 1

 Description   

PrimaryOnlyService::lookupInstance() invariants a list of various conditions in order to prevent a possible deadlock during the case described in SERVER-51650. This invariant currently fails if lookupInstance() is called during a lock-free read which holds the global lock in mode IS as seen in BF-24817. Since a lock-free read doesn't hold any locks which would prevent a step down, they should be excepted from the invariant.



 Comments   
Comment by Max Hirschhorn [ 12/Apr/22 ]

What are the use cases under which we need to be acquiring a POS while holding a lock?

Thanks kaloian.manassiev@mongodb.com, I can't speak for all use cases where we're holding locks but the two we've had come up in resharding are:

  1. Updating the in-memory state of the corresponding ReshardingCoordinator instance on storage transaction commit when a participant shard writes to the config.reshardingOperations collection. (SERVER-61482, SERVER-49572)
  2. Updating the in-memory state of the corresponding DonorStateMachine instance to track when a read or write operation came in while the sharding critical section is held for the namespace. (SERVER-64372)

Maybe for (2) it would be preferable to increment the counter in service_entry_point_common.cpp after locks, storage engine snapshots, etc. have already been released.

An alternative to (1) which I think Esha had proposed in the past would be to have a decided command for participant shards to run on the config server primary to update the config.reshardingOperations collection and then not as an OpObserver update the ReshardingCoordinator instance in-memory state. As part of the Global Index Coordinator we can see about doing away with some of the complex work which happens in an OpObserver.

Comment by Kaloian Manassiev [ 12/Apr/22 ]

What are the use cases under which we need to be acquiring a POS while holding a lock? The POS reads from disk, has thread management and so on. Therefore we should not be acquiring it while holding locks or while on the path of a CRUD operation. In addition to locks, LFR could also be holding storage engine snapshots (which is effectively locks just at some lower level), acquiring the POS does the same, therefore there are subtle circular dependencies that are not expressed in terms of the lock graph.

CC max.hirschhorn@mongodb.com

Generated at Thu Feb 08 06:02:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.