-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical - P2
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines - Server Integration
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The problem
A brand-new (fresh) disaggregated-storage cluster can come up dead on arrival. In HELP-98471, two dev clusters (sls-smoke-dev-aws-euc1, sls-canva-docs-usw2) after the rc1019 upgrade were stuck in STARTUP2 with no primary; ~5% of create-and-delete cluster runs also time out on creation (Aaron Himelman, 09-04).
On the node that won the first election (atlas-htj1iu-shard-00-01, 2026-08-14 05:44 UTC), the step-up thread:
- committed the term-1 no-op to the log service (11047205 — the point of no return),
- promoted the storage engine to leader, opened the catalog, created the oplog, wrote the no-op (11783101 / 11379200 / 10985462),
- completed the fresh-cluster KEK init (12623633 — the last log line the thread ever produced),
- and then produced no further log for 2h35m until automation killed it; FCV stayed unset. It never took the cluster's first checkpoint.
Because the term-1 entry is already in the log service but no checkpoint exists, every other node fails step-up with the 12631111/12631113 "no prior primary has written a checkpoint" escalation and the cluster stays in STARTUP2. This is unrecoverable-by-design: recreation is the only remediation, and a recreated cluster can hit the same state again.
Where the thread stops (what the code says)
The last statement before the stall is the FCV-init global lock in the fresh-cluster step-up path:
Lock::GlobalLock(opCtx, MODE_X, Date_t::max(), kThrow, {skipRSTLLock, Intent::Write}) (sls_state_machine/state_utils.cpp)
Several properties of that acquisition, combined, make a real wedge window:
- On the fresh path (no oplog / no checkpoint) the step-up does not hold the Global X fence that the non-fresh path takes and holds across the whole critical section — the code path that would take that lock is skipped entirely for a node with no oplog.
- The fresh path instead takes Global X twice inside finalizeStepUp, both after the AppendLog point of no return, both with Date_t::max() (no deadline).
- The step-up's intent-registry interruption (killConflictingOperations with InterruptionType.StepUp) kills and drains nothing, and read/local-write intents remain admissible while the step-up is in progress; because intent registration replaced RSTL acquisitions, the step-up's RSTL no longer fences readers.
- The global lock is a fair queue: a single holder that does not release blocks a MODE_X waiter indefinitely, with no timeout and nothing to interrupt or kill the waiter.
Net: if any operation acquires and holds a conflicting global lock after the term-1 AppendLog commit and before the FCV Global X, the step-up waits on it forever. The node is wedged mid-step-up, no first checkpoint is ever taken, and the cluster cannot form.
Evidence
- Incident: HELP-98471 (dev). Holder identity was never captured — there is no lockInfo output and no stack trace from the stuck window; the diagnostics uploaded to the ticket post-date the recreation. "Who held the lock" remains unconfirmed.
- Controlled reproduction (2026-09-09, passes under the disagg_storage suite): step_up_hangs_when_global_lock_held.js with a test failpoint placed at the stall point. It holds a global MODE_IS lock in the window (between the AppendLog commit, 11047205, and the FCV Global X), resumes the step-up, and observes: the node stays not-primary for the entire hold (the step-up is blocked), then becomes primary and takes the first checkpoint immediately after the holder releases. The only difference from a clean fresh_cluster_step_up run is the presence of the holder, so the global lock is demonstrated to be the blocker. The test can be flipped to "must not wedge" once a fix bounds or avoids the unbounded acquisition.
Relationship to SERVER-134279 (HELP-99259) — a sibling, not the same defect
SERVER-134279 is the same problem class but a different mechanism:
- SERVER-134279 (HELP-99259): the step-up wedges on the Global X fence (never acquires it), with a deadlock cycle whose keystone is the Checkpointer mid-checkpoint-load at standby teardown; the holders are reader operations parked inside WiredTiger.
- This ticket: the fresh-cluster path skips that fence entirely, and the step-up wedges on the later FCV-init Global X behind an ordinary global-lock holder. The reproduction shows the wedge needs no checkpoint/teardown cycle at all — a single non-releasing holder is sufficient.
Consequence: a fix scoped to the 134279 cycle would not necessarily cover this path; any fix should be checked against both. Note also that the absence of log 11313304 ("Acquired Global X for step up") appears in BOTH incidents for different reasons — in 134279 the fence lock was never acquired; on the fresh path it is skipped by design — so that marker is not a reliable discriminator on its own.
Also related: SERVER-125948 / SERVER-126740 cover the general problem "stepUpIfEligible can hang indefinitely and is not interruptible"; this ticket is a concrete instance with a passing reproducer, and should be checked against that work rather than fixed in isolation.
What is needed
- An explanation of who holds the conflicting global lock in the window on a real occurrence (no lockInfo/stack was ever captured), i.e. confirmation of the production holder.
- A decision on whether a no-deadline Global X acquisition in the step-up path, after the point of no return, should be bounded (in time) or fenced from before the point of no return — the same question SERVER-134279 raises, now shown to apply to the fresh-cluster path as well.
Related: SERVER-134279 (HELP-99259) — sibling class, different mechanism; SERVER-125948 / SERVER-126740 — the stepUpIfEligible interruptibility umbrella (this is a concrete instance); HELP-98471 — this incident.
- related to
-
SERVER-134279 Step-up deadlocks when the Checkpointer is mid-checkpoint-load at standby teardown; no deadline or recovery path bounds the outage
-
- Open
-