Quiesced resharding operations can cause conflicts during retries of ongoing operations

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.0.0-rc2
    • Affects Version/s: 7.2.0, 8.0.0, 9.0.0-rc0, 8.3.0
    • Component/s: None
    • None
    • Cluster Scalability
    • Fully Compatible
    • ALL
    • v9.0, v8.0
    • ClusterScalability 20Jul-3Aug
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      For a resharding operation with a user specified UUID, that operation will remain in a quiesce state for reshardingCoordinatorQuiescePeriodMillis which defaults to 15 minutes. While an operation is in this state, a failover of the config server will cause the coordinator to be "re-executed" in order to mark it again as completed and waiting for quiesce due to the step up process of the PrimaryOnlyService.

      A quiesced resharding operation will take this fast path to abort actually running the coordinator again, which sets the completion future on the coordinator instance. This re-execution, however, introduces a window of time in which this operation can conflict with new _configsvrReshardCollection commands since the completion future is not yet marked as ready again. This is a problem if there is also another resharding operation ongoing and a concurrent failover on the dbPrimary shard for the collection being resharded because the _configsvrReshardCollection command which hits the conflict is for an already running resharding operation.

      Thus we have the following interleaving:

      1. Run a resharding operation with a user UUID and have it enter quiesce mode
      2. Start a new resharding operation, it is ongoing
      3. Step down the config server
      4. Before the quiesced resharding operation marks the completion future as ready again, step down the dbPrimary for the newly ongoing resharding operation
      5. Upon step up, the new primary on the dbPrimary shard will re-send _configsvrReshardCollection for the ongoing resharding
      6. This command will conflict with the quiesced resharding operation and return ReshardCollectionInProgress
      7. The dbPrimary will exit the DDL coordinator and release the DDL locks
      8. The config server completes the recovery of the resharding coordiantors and is now executing the new resharding operation while no DDL locks are held

            Assignee:
            Kruti Shah
            Reporter:
            Allison Easton
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: