-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Transactions
-
None
-
Storage Engines - Transactions
-
193.446
-
SE Transactions - 2026-10-23
-
1
Issue Summary
The deferred checkpoint pick-up server on a disaggregated follower only uses a timed wait while a checkpoint is queued. With nothing queued it waits with no timeout. If a wakeup is ever missed in that state, the server stays asleep with no backstop, and a checkpoint that gets deferred around that time could be stranded until something else signals it. The server's wakeups should always have a timeout.
Context
- [SERVER-136904]: a deferred checkpoint on the handoff target was adopted exactly 10 seconds after delivery, which matches the pick-up server's timer. It is not yet known whether a snapshot really blocked for 10 seconds or a wakeup was missed and the timer found the work.
WT-18908adds statistics to tell these apart, including a count of timer wakeups that found an adoptable checkpoint. - The server decides how long to sleep by looking at the queue just before it waits, without a lock. A deferral or a pin release that signals between that look and the wait is only remembered by the wait primitive in the common case. The wait primitive has a very small window, between a waiter announcing itself and taking the mutex, where a signal reaches nobody and is not remembered. With a timed wait that costs up to the timer interval; with an untimed wait it can cost indefinitely.
- The timer is currently a backstop for the case where something is queued. When nothing is queued the wait is untimed, so the case where a deferral races with the server going to sleep has no backstop at all.
- The 10 second interval is long compared with the planned step-down deadline, which is also 10 seconds. A missed wakeup that costs one full interval is enough to fail a step-down with handoff (see SERVER-137084 for the server-side change that removes that dependence).
Proposed Solution
- Make the pick-up server's wait always timed, whether or not anything is queued, so a missed wakeup in any state is recovered by the next timer wake.
- Use a shorter interval while a checkpoint is queued, so a missed wakeup costs about a second instead of ten. Each wake while something is queued does a scan of the session array for the oldest pinned generation and takes the queue lock for a short time, which is cheap at one wake per second and only happens while a checkpoint is deferred.
- Keep a longer interval, or the same one, while nothing is queued. An idle wake only looks at an empty queue.
- Do not go below about one second: a shorter interval adds wakeups without improving the bound in a way that matters, since adoption itself takes tens of milliseconds.
- Decide the intervals using the
WT-18908statistics once they have run in CI. If the timer never finds adoptable work, the shorter interval is only insurance. If it does, find and fix the missed wakeup itself rather than relying on the timer. - Keep the comment on the wait explaining that the timer is a backstop and not an adoption deadline.
Definition of Done
- The pick-up server never waits without a timeout.
- A test checks that a deferred checkpoint is still adopted, within the shorter interval, when its wakeup is suppressed (for example by a debug hook that skips the signal), and that adoption stays prompt when the signal does arrive.
- No measurable change in idle CPU use of a follower with nothing queued.
- related to
-
WT-18908 Improve disaggregated checkpoint pick-up statistics to show what blocks a deferred adoption
-
- Closed
-