While executing SERVER-127622 we discovered that movePrimary triggers a tripwire when checking for consistency of in-memory and disk states if setFCV and migrations are happening simultaneously.

This is because the following assumptions are made for movePrimary and setFCV:

• setFCV:
  • The cloner finishing means all shards can be considered with authoritative information
• movePrimary:
  • If it starts during kUpgrading then it assumes no migrations can occur on the two shards

However, those assumptions are broken if setFCV, movePrimary, and migrations are occurring simultaneously as the following can happen during a setFCV timeline of operations:

• The cloner may not trigger a clone on a database that's undergoing a movePrimary as it expects the movePrimary to make the database authoritative on the shards.
• Once the cloner finishes it lets the config server proceed with the switch to kFullyUpgraded on all shards.
  • Crucially this step does not await the potential movePrimary in kUpgrading to finish so at this point we're transitioning all shards to say they can be fully authoritative yet the metadata hasn't been fully copied to the shards.
• Migrations are enabled gradually on all the shards as they move to kFullyUpgraded.
• At this point a migration could affect one of the two nodes in a movePrimary breaking the assumption that during kUpgrading no migrations can happen from the perspective of the coordinator.