ISSUE DESCRIPTION AND IMPACT

This issue can cause unavailability of a shard in sharded clusters running MongoDB versions 5.0.0 - 5.0.5 and 5.1.0 - 5.1.1. Next versions are not affected.

The problem can potentially occur if all of the following conditions have been met at least once:

• More than one sharded collection
• Multiple migrations
• Intense write workloads or hardware failures

Symptom of the bug: mongod process crashing upon step-up due to an invariant failure with the following message: "Upon step-up a second migration coordinator was found".

REMEDIATION AND WORKAROUNDS

• Restart nodes of the shard as replica set
• Double-check that at most one migration coordinator document does not have a definitive decision.
• For each migration coordinator document with a definitive decision, double-check that range deletion tasks are consistent with migration coordinators (same range and collectionUUID, if present):
  • Aborted decision:
    — No range deletion document on donor
    — Zero or one ready range deletion document on recipient
  • Committed decision:
    — Zero or one ready range deletion document on donor
    — No range deletion document on recipient
  • No decision:
    — One pending range deletion tasks on donor
    — One pending range deletion tasks on recipient
• Majority-delete all migration coordinators with a definitive decision
• Restart nodes as shard

TECHNICAL DETAILS

Migration coordinators:

• Documents persisted locally on shards in the internal collection config.migrationCoordinators 
• The structure of migration coordinator documents can be found here.

Range deletion tasks:

• Documents persisted locally on shards in the internal collection config.rangeDeletions

• The structure of range deletion task documents can be found here