(Test only issue)

Issue

There is a potential deadlock between checkMetadataConsistency on secondaries and rollback, as follows:

1. [ROLLBACK] A rollback starts.
2. [CMC] A _shardsvrCheckMetadataConsistencySecondaryParticipant command arrives and starts running.
3. [CMC] Inside checkCollectionMetadataInShardCatalog, it calls into checkCatalogStability, which acquires the CSR lock.
4. [CMC] Still inside checkCollectionMetadataInShardCatalog with the CSR lock held, it calls into getShardCatalogRelaxedChunkInfo, which will do a snapshot read via DBDirectClient.
5. [ROLLBACK] Calls into ShardingRecoveryService::onReplicationRollback, then ShardingRecoveryService::_resetInMemoryStates, which tries to acquire the CSR lock.

At this point the checkMetadataConsistency is waiting for the rollback to finish to satisfy the read concern, and the rollback is waiting for the CSR lock, so we are deadlocked.

Note

Rollback kills running operations, so it is not possible to reverse steps (1) and (2) - the _shardsvrCheckMetadataConsistencySecondaryParticipant must start after rollback.

In particular, with SERVER-132105, this particular deadlock doesn't seem to reproduce, since _shardsvrCheckMetadataConsistencySecondaryParticipant can't start during rollback in step (2).

Fix

We should immediately release the CSR lock (similar to how it's done below) to avoid calling into DBDirectClient with locks held.