-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines - Foundations
-
31.232
-
None
-
None
WT-18598 added a bounded wait to the leader/follower rendezvous in test/format multi-node runs so that a stalled process fails fast with diagnostics instead of hanging until the two-hour Evergreen idle timeout. The bound was first set to two minutes, which turned out to be far too tight: runs on ubuntu2004-asan aborted while the follower was simply working through its end-of-run validation. Raising the bound to ten minutes was expected to leave a comfortable margin, but a release build (ubuntu2004-release-nonstandalone, format-stress-test-disagg-multi-2) still hit it — the leader reached the sync point, waited ten full minutes and aborted while the follower had not arrived. The bound has now been raised to thirty minutes so it can serve its intended purpose as a deadlock guard rather than a lag policy.
The remaining thirty-minute bound is a workaround, not an answer. A follower trailing the leader by more than ten minutes on a release build, with no sanitizer overhead in play, is not normal pacing; the leader has already finished its operations, its checksum walk and its rollback-to-stable check by the time it starts waiting. Something on the follower side — checkpoint pickup, stable dhandle handling, or the end-of-run verification path — is taking far longer than the workload should require, and it may be the same underlying slowness that produced the original silent hangs. This ticket is to reproduce the lag, measure where the follower actually spends that time, and decide whether it is a genuine performance bug or an artifact of how the multi-node test drives the follower.