Consider the case where we have two nodes, A and B, where A is the primary. If we call movePrimary to move A to B and at the same time call removeShard on B, then the following might happen.
- movePrimary copies unsharded collections from A to B.
- removeShard checks if B is the primary, which it is not.
- movePrimary updates the config server, which now thinks B is the primary.
- removeShard removes B.
- movePrimary deletes original unsharded collections from A.
If the above were to happen, then all unsharded collections are deleted, and there would exist no primary, which is very bad.
An FSM concurrency workload should be written to test this type of behavior, and what would happen.