CI failures

Evergreen's disaggregated-storage multi-node format tasks (format-stress-test-disagg-multi-1/-2 and variants) intermittently hang until the two-hour idle watchdog kills them as task-timed-out. Between 2026-09-04 and 09-07 there were nineteen such failures across arm64/x86, ASan/release, and standalone/nonstandalone variants, all duplicated to WT-18245. None had a failing test, so these are pure hangs — the format processes stop making progress and stop printing — rather than correctness errors. The intermittency points to a randomly-enabled test option.

The problem

The WT hang-analyzer stacks show the two processes stuck at the leader/follower barrier: the leader finished its operations loop and blocks in a timeout-free read() in disagg_multi_sync_point (format_disagg.c), while every follower worker sleeps one second in __session_dhandle_stable_delay_stress opening a stable-checkpoint cursor. That sleep is the disagg_stable_dhandle_delay stress knob from WT-18532, which fires only for stable URIs — in multi-node only the follower opens those, so it slows the follower by about a second per cursor open while the leader runs at full speed. The barrier read() has no timeout and sits outside format's own abort guard, so a starved follower leaves the leader blocked forever and output-less until Evergreen's two-hour idle timeout fires. The knob is randomly enabled (~2% of runs), which matches the intermittent pattern; the earlier 2026-07-30 instances predate it and reflect the same unbounded-barrier gap.

Other test/format modes are unaffected because the one-second stall only ever delays the single process that opens stable-checkpoint cursors — only multi-node adds both a follower that pays the stall and a leader blocked at a timeout-free barrier waiting on it, which is what turns a slowdown into a hang.