Problem statement

During disagg step-up, the stepping-up thread cancels the standby log-server loop and joins the log-server-manager main thread before acquiring the Global X lock. If, at that instant, any operation holding a global intent lock is parked inside the storage layer on work that needs disagg progress, the node deadlocks:

• the step-up's Global X waits behind the intent lock (the Global lock is a fair queue);
• the intent-lock holder is parked inside the storage layer where the step-up's op-kill interrupt is never polled;
• the storage progress it needs requires the standby machinery the step-up just tore down, which returns only when the step-up completes.

What is proven vs. hypothesized in the observed occurrence: that some global intent lock was held continuously for the full 3.5-day wedge is proven (the Global lock is a fair queue; the X was never granted; the same node's 2,263 immediately preceding step-up attempts each acquired Global X in 0ms). The holder's identity is a ranked hypothesis: most plausibly a command thread parked uninterruptibly in the WT follower read path, serialized behind the Checkpointer thread's in-flight checkpoint load (the Checkpointer — normally active every 60s — logged a checkpoint-load read 2.2s before the standby teardown and then nothing for 2.5 days; WT's sweep server reported sessions stuck for 60+ minutes an hour in; zero page-read log lines after the wedge places the park before the page log). Confirming the holder requires lockInfo/all-thread stacks from a recurrence — see the runbook section.

Nothing bounds the wait: the Global X acquisition has no deadline, fassertOnLockTimeoutForStepUpDown covers only the RSTL, and the election timer is permanently short-circuited by the alreadySteppingUp guard. The node stays in this state indefinitely while continuing to accept TCP/TLS connections, so it looks alive to network-level health checks.

Production occurrence (HELP-99259, dev/euc1/sls-unruly-archives-06 00-01)

Node 00-01 (binary 9.0.0-alpha0, git 77fb68384) deadlocked this way for 3 days 14 hours (2026-08-27T17:12:21Z -> 2026-08-31T07:33Z, ended only by the dsc6 upgrade restart). From the mongod log:

17:12:19.548 Checkpointer  "Loading checkpoint: root_id=923 flags=0 lsn=7678357367158742247 ..."
             (WT_CURSOR.next on collection-5ca84ad0...wt_stable/WiredTigerCheckpoint.816)
             <- Checkpointer's last log line ever
17:12:21.780 Disagg-0      "Catching up for step-up succeeded" ... applier drained
17:12:21.781 Disagg-0      "Canceling the log server standby loop."
17:12:21.781 Disagg-0      "Joining log server manager main thread."
17:12:21.797 Disagg-0      "Acquired RSTL for step up" (6ms)
             <- Disagg-0's last log line ever; 11313304 "Acquired Global X for step up" never logged

For contrast, the node's 2,263 immediately preceding step-up attempts (it was crash-looping every ~13s on the WT-18473 drain assertion) each logged "Acquired Global X" 0ms after the RSTL line. Whether an attempt crashed (reached the drain) or deadlocked (parked at Global X) was a pure timing race against the Checkpointer.

Consequences observed while wedged:

• Election timer fired every 2s for 3.5 days (~150k firings), every one short-circuited by "Node is already stepping up" (11885301). The recovery mechanism is structurally defeated exactly when needed.
• ThroughputProbing logged a continuous stall; zero operations reached storage; only failing monitoring commands "completed".
• Connections leaked linearly (~116/hour) to the 9,086 cap; once the peer went down, the client reconnect stampede drove 3.7M pthread_create EAGAIN failures (08-31 02:39-07:28Z).
• An automation-initiated restart (08-31 03:45Z) never took effect against the wedged process.

Relation to prior incidents

• HELP-99002: same signature (step-up parked between RSTL and Global X after tearing down its own standby loop; 28-minute outage). The circularity was hypothesized there; this occurrence confirms the shape of the cycle and identifies its keystone (the Checkpointer's in-flight checkpoint load at teardown time), with the intent-lock holder's exact identity still to be confirmed from stacks (see runbook).
• HELP-98471: step-up silently parked on an unbounded Global X on the fresh-cluster path.
• The trigger that put the node in a step-up loop was WT-18473 (fixed in dsc6), but the deadlock is independent of that trigger: any step-up racing a checkpoint load can hit it on current code.

Candidate directions (non-binding)

• Bound the step-up Global X acquisition (fail fast instead of wedging silently).
• Reorder the standby-machinery teardown relative to the lock acquisition.
• Give the alreadySteppingUp guard a deadline so the election timer remains a recovery path.
• Make storage-layer waits on the step-up path observe interruption.

Next-occurrence runbook (how to capture what the logs cannot)

The WT-18473 fix removes the trigger that made this occurrence likely, so recurrences will be rare — capture must be first-try:

• Before any kill or restart of a suspected-wedged node: {{db.adminCommand( {lockInfo: 1}

  )}} on the node (names the Global-lock holder directly). If commands cannot be admitted, use the priority/localhost port.

• kill -SIGUSR2 <pid> for all-thread stacks (CLOUDP-439973 is automating this); the stack of interest is any thread inside the WT read/dhandle-open path below a command or Checkpointer frame.
• Detection signature for automation: node in SECONDARY logging 11885301 ("Node is already stepping up") repeatedly for more than a few minutes with no 11313304 since the matching 11148202 ("Acquired RSTL for step up").

Evidence / links

• HELP-99259 — this incident; full log analysis in the ticket comments
• HELP-99002, HELP-98471 — same defect class
• WT-18473, AF-20552 — the crash-loop trigger, fixed in dsc6 via mongo commit c4628d0e010
• Logs: sls-unruly-archives-06-shard-00-01, segments 2026-08-27T00:00Z-08-30T06:00Z (onset) and 2026-08-30T16:06Z-08-31T16:11Z (wedge tail + recovery)