Issue Summary

When a follower defers adopting a delivered checkpoint because an older transaction snapshot is still active, the existing statistics cannot tell an operator whether anything is blocking the adoption, how many sessions are blocking it, or how long it has been waiting. Diagnosing a stalled or slow adoption currently needs a core dump taken at the right moment.

Context

• [SERVER-136904]: a planned step-down failed because the handoff target adopted the step-down checkpoint exactly 10 seconds after delivery. The only existing signal was that one pick-up was deferred and that the adopted checkpoint LSN trailed the delivered one. FTDC (one sample per second) showed a deferral and a 10 second wait, but could not show whether a snapshot was really blocking for 10 seconds or whether the pick-up server missed a wakeup and was rescued by its 10 second timer. The only core dump was taken from the wrong node about 40 minutes after the failure.
• Existing statistics: pick-ups deferred (counter), most recently delivered LSN, most recently adopted LSN, snapshots rebuilt after racing a pick-up, and step-up deferred pick-up retries.
• Related: WT-18156 introduced the deferral queue.
• Observability on snapshot begin/release is hot path. It must not add locks or atomics there.

Proposed Solution

A prototype exists on a diagnostic patch and computes the first three items when statistics are read, so it adds nothing to the hot path:

• Checkpoints delivered and awaiting adoption (gauge).
• Wait time of the oldest deferred checkpoint, in milliseconds (gauge).
• Number of sessions whose snapshot blocks the oldest deferred checkpoint (gauge).

Further items to evaluate:

• Age of the oldest blocking snapshot. Needs a start time on the session; only add it if it can be recorded without touching the snapshot hot path.
• Pick-up server wakeups split by reason: signalled by a snapshot release or a deferral, versus the timed backstop. Plus a counter of backstop wakeups that found an adoptable checkpoint (expected to always be zero, non-zero means a missed wakeup).
• Time from the last blocker releasing to the adoption completing, kept as a histogram or most recent and maximum values.
• A rate limited warning once a deferral has waited longer than a threshold, naming the blocking sessions (session id, transaction id, snapshot age) so the blocker can be identified without a core dump.
• Decide whether the 10 second backstop should always run, even with nothing queued, and whether it should be configurable.

Definition of Done

• New statistics appear in the connection statistics output (and therefore in FTDC) and are documented in the statistics data definitions.
• A test blocks a deferred adoption with one and with several snapshots and checks queue length, blocking session count and wait time before and after release.
• A test checks that adoption after the last blocker releases completes well under the backstop interval, so a missed wakeup fails the test instead of being hidden by the 10 second timer.
• No measurable change in snapshot begin/release cost.