When enabling the authoritative shards feature as part of an FCV upgrade, setFCV will tell every shards to clone the authoritative metadata.  To do this, it first grabs the shard membership lock to get a stable shard list, then it sends a command to each shard to start a DDL coordinator.

This is problematic because removeShard, which can run concurrently to an FCV upgrade, first blocks DDL coordinators, then it grabs the shard membership lock, in effect locking in the opposite order. Thus, they can interleave such that both setFCV and removeShard deadlock.

A reproducer is attached. This issue exists with both the legacy removeShard (upgrade from v8.0 to v8.3), and the new removeShard coordinator (upgrade from v8.2 to v8.3). It could also be a problem with other FCV upgrade features if they spawn DDL coordinators similarly.