Fresh-cluster first step-up can wedge forever after committing the term-1 entry to the log service, leaving a brand-new cluster stuck in STARTUP2 and unrecoverable

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Engines - Server Integration
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The problem

      A brand-new (fresh) disaggregated-storage cluster can come up dead on arrival. In HELP-98471, two dev clusters (sls-smoke-dev-aws-euc1, sls-canva-docs-usw2) after the rc1019 upgrade were stuck in STARTUP2 with no primary; ~5% of create-and-delete cluster runs also time out on creation (Aaron Himelman, 09-04).

      On the node that won the first election (atlas-htj1iu-shard-00-01, 2026-08-14 05:44 UTC), the step-up thread:

      • committed the term-1 no-op to the log service (11047205 — the point of no return),
      • promoted the storage engine to leader, opened the catalog, created the oplog, wrote the no-op (11783101 / 11379200 / 10985462),
      • completed the fresh-cluster KEK init (12623633 — the last log line the thread ever produced),
      • and then produced no further log for 2h35m until automation killed it; FCV stayed unset. It never took the cluster's first checkpoint.

      Because the term-1 entry is already in the log service but no checkpoint exists, every other node fails step-up with the 12631111/12631113 "no prior primary has written a checkpoint" escalation and the cluster stays in STARTUP2. This is unrecoverable-by-design: recreation is the only remediation, and a recreated cluster can hit the same state again.

      Where the thread stops (what the code says)

      The last statement before the stall is the FCV-init global lock in the fresh-cluster step-up path:

      Lock::GlobalLock(opCtx, MODE_X, Date_t::max(), kThrow, {skipRSTLLock, Intent::Write}) (sls_state_machine/state_utils.cpp)

      Several properties of that acquisition, combined, make a real wedge window:

      1. On the fresh path (no oplog / no checkpoint) the step-up does not hold the Global X fence that the non-fresh path takes and holds across the whole critical section — the code path that would take that lock is skipped entirely for a node with no oplog.
      2. The fresh path instead takes Global X twice inside finalizeStepUp, both after the AppendLog point of no return, both with Date_t::max() (no deadline).
      3. The step-up's intent-registry interruption (killConflictingOperations with InterruptionType.StepUp) kills and drains nothing, and read/local-write intents remain admissible while the step-up is in progress; because intent registration replaced RSTL acquisitions, the step-up's RSTL no longer fences readers.
      4. The global lock is a fair queue: a single holder that does not release blocks a MODE_X waiter indefinitely, with no timeout and nothing to interrupt or kill the waiter.

      Net: if any operation acquires and holds a conflicting global lock after the term-1 AppendLog commit and before the FCV Global X, the step-up waits on it forever. The node is wedged mid-step-up, no first checkpoint is ever taken, and the cluster cannot form.

      Evidence

      • Incident: HELP-98471 (dev). Holder identity was never captured — there is no lockInfo output and no stack trace from the stuck window; the diagnostics uploaded to the ticket post-date the recreation. "Who held the lock" remains unconfirmed.
      • Controlled reproduction (2026-09-09, passes under the disagg_storage suite): step_up_hangs_when_global_lock_held.js with a test failpoint placed at the stall point. It holds a global MODE_IS lock in the window (between the AppendLog commit, 11047205, and the FCV Global X), resumes the step-up, and observes: the node stays not-primary for the entire hold (the step-up is blocked), then becomes primary and takes the first checkpoint immediately after the holder releases. The only difference from a clean fresh_cluster_step_up run is the presence of the holder, so the global lock is demonstrated to be the blocker. The test can be flipped to "must not wedge" once a fix bounds or avoids the unbounded acquisition.

      Relationship to SERVER-134279 (HELP-99259) — a sibling, not the same defect

      SERVER-134279 is the same problem class but a different mechanism:

      • SERVER-134279 (HELP-99259): the step-up wedges on the Global X fence (never acquires it), with a deadlock cycle whose keystone is the Checkpointer mid-checkpoint-load at standby teardown; the holders are reader operations parked inside WiredTiger.
      • This ticket: the fresh-cluster path skips that fence entirely, and the step-up wedges on the later FCV-init Global X behind an ordinary global-lock holder. The reproduction shows the wedge needs no checkpoint/teardown cycle at all — a single non-releasing holder is sufficient.

      Consequence: a fix scoped to the 134279 cycle would not necessarily cover this path; any fix should be checked against both. Note also that the absence of log 11313304 ("Acquired Global X for step up") appears in BOTH incidents for different reasons — in 134279 the fence lock was never acquired; on the fresh path it is skipped by design — so that marker is not a reliable discriminator on its own.

      Also related: SERVER-125948 / SERVER-126740 cover the general problem "stepUpIfEligible can hang indefinitely and is not interruptible"; this ticket is a concrete instance with a passing reproducer, and should be checked against that work rather than fixed in isolation.

      What is needed

      • An explanation of who holds the conflicting global lock in the window on a real occurrence (no lockInfo/stack was ever captured), i.e. confirmation of the production holder.
      • A decision on whether a no-deadline Global X acquisition in the step-up path, after the point of no return, should be bounded (in time) or fenced from before the point of no return — the same question SERVER-134279 raises, now shown to apply to the fresh-cluster path as well.

      Related: SERVER-134279 (HELP-99259) — sibling class, different mechanism; SERVER-125948 / SERVER-126740 — the stepUpIfEligible interruptibility umbrella (this is a concrete instance); HELP-98471 — this incident.

            Assignee:
            Unassigned
            Reporter:
            Daotang Yang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: