As part of the work on SERVER-127622 we tried to expand the FSM coverage of DDLs with CRUD operations as well as potential balancer operations going in the background.

This revealed a deadlock that can occur between resharding, chunk migrations, AND setFCV.

The following is the sequence of events that leads to the deadlock in question:

• The config server attempts to create the resharding coordinator on the CSRS. This is done under a FixedFCVRegion.
• As part of the coordinator creation an aggregation is sent to the shards in order to determine an estimated data size.
• In parallel the chunk migration attempts to finish and enters the critical section before going to commit the migration on the CSRS.
• The aggregation as a result waits until the chunk migration commits since the critical section is active.
• A setFCV operation occurs and attempts to update the FCV document, to do this it requires an exclusive lock on the FCV resource that FixedFCVRegion targets. This blocks behind the resharding coordinator FCV lock and prevents other operations coming after.
• The chunk migration tries to commit on the CSRS and to do so requires a FixedFCVRegion.

At this point the stage is set for the deadlock, as the following lock dependency graph is in place: resharding -> chunk migration -> setFCV -> resharding.