StepUp hangs when a stale in-memory chunk operation blocks recovery of a persisted recipient

XMLWordPrintableJSON

    • Catalog and Routing
    • Fully Compatible
    • ALL
    • v9.0
    • CAR Team 2026-08-17, CAR Team 2026-08-31
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      A secondary node can hold a stale ActiveMigrationsRegistry slot, blocking step-up when the shard needs to recover a persisted chunk migration as a recipient.

      Why can a secondary node retain a stale ActiveMigrationsRegistry slot?

      Chunk-operation coordinators (moveRange, split, and merge) acquire an ActiveMigrationsRegistry slot. The slot is released either when the coordinator instance is destroyed or when _releaseLocks() is called by the .onCompletion(...) task scheduled at the end of a ShardingCoordinator task chain.

      Two conditions lead to this scenario:

      • During step-down, an ongoing coordinator may skip its .onCompletion(...) task. As part of the step-down routine, the PrimaryOnlyService shuts down the scoped executor, which unschedules all pending tasks in that executor. Therefore, if the executor shuts down before the .onCompletion(...) task starts, the coordinator does not release its ActiveMigrationsRegistry slot during step-down.
      • The PrimaryOnlyService destroys active coordinator instances only during step-up. Consequently, if the ActiveMigrationsRegistry slot is not released as part of the task chain, it remains acquired until PrimaryOnlyService::onStepUp() is called.

      As a result, a secondary node can retain an ActiveMigrationsRegistry slot under a ShardingCoordinator instance.

      Why does step-up hang?

      1. During step-up, if config.migrationRecipients contains a document, the sharding recovery routine recovers the persisted recipient migration. This recovery runs before PrimaryOnlyService::stepUp().
      2. Recipient migration recovery attempts to acquire an ActiveMigrationsRegistry slot. The acquisition waits without a deadline until the required slots become available.
      3. The wait cannot complete because the slots are released only when the PrimaryOnlyService later joins the leftover work, which is the next step in the same step-up sequence.

      As a result, step-up hangs, leaving the shard unavailable.

            Assignee:
            Silvia Surroca
            Reporter:
            Pol Pinol
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: