ReshardCollectionCoordinator can incorrectly restart a reshardingCoordinator after a failover

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Cluster Scalability
    • ALL
    • v9.0
    • Cluster Scalability Priorities
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      TL;DR An aborted or committed resharding operation can end up restarting on retry if the db-primary shard fails over (or has full restart, like magic restore).

       

      Resharding has two coordinators:

      1. ReshardCollectionCoordinator: runs on the db-primary shard and sends _configsvrReshardCollection.
      2. ReshardingCoordinator: runs the config server and does the orchestration.

       

      Due to this structure, the following sequence can happen:

      • ReshardCollection starts; ReshardCollectionCoordinator sends _configsvrReshardCollection and waits for its completion
      • ReshardingCoordinator begins
      • AbortReshardCollection is invoked
      • ReshardingCoordinator finishes aborting and removes its state doc
      • The db-primary shard fails over before ReshardCollectionCoordinator gets signal from _configsvrReshardCollection.
      • On stepup, ReshardCollectionCoordinator re-sends _configsvrReshardCollection.
      • With nothing to join, a fresh ReshardingCoordinator instance gets created and runs all the way to a successful commit.

       

      The result is that an operation the user aborted completes anyway, changing the collection's shard key and UUID. Or on the commit case, redo the operation if forceRedistribution is specified.

      Note: resharding operations that pass a user-supplied reshardingUUID aren't affected, because they quiesce the state document on abort instead of deleting it, so the retry has a record to dedup against.

        1. SERVER-131275_optionb.pdf
          112 kB
        2. diff_bf44354.diff
          8 kB

            Assignee:
            Unassigned
            Reporter:
            Kruti Shah
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: