Concurrent movePrimary and startShardDraining/removeShard can cause movePrimary to hang

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • 🟥 DDL
    • None
    • None
    • None
    • None
    • None
    • None

      The movePrimary coordinator has initial checks which are done outside of a phase here. This means that these will be re-executed rather than skipped if the coordinator retries after reaching a later phase. The coordinator also has mustAlwaysMakeProgress enabled from any phase other than unset. This allows for the following scenario:

      1. MovePrimary starts from shard 0 to shard 1 and enters some phase other than unset
      2. StartShardDraining or RemoveShard is called on shard 1, causing the shard to enter draining
      3. MovePrimary hits some error, the coordinator is in must always make progress so it retries
      4. The initial checks are re-executed and see shard 1 in draining, throwing shardNotFound
      5. Since the coordinator is in mustAlwaysMakeProgress, this will continue until the draining flag is removed from the destination shard (note that successfully removing the shard would not work even if it were possible because there is also a precondition check that would fail in this case)

        1. diff.patch
          4 kB
          Allison Easton

            Assignee:
            Unassigned
            Reporter:
            Allison Easton
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: