Summary

We do not yet understand what drives the longer step-down checkpoint durations observed under load in HELP-101625. Available logs and FTDC provide insufficient detail to explain the additional time compared with routine checkpoints.

Current behaviour

HELP-101625 reports step-down checkpoints taking approximately 11–27 seconds under load, compared with reported routine checkpoint durations of 5.2–11.8 seconds. Five of six planned handoffs aborted after checkpoint completion because the 10-second handoff deadline had expired.

Checkpoint preparation completed quickly in the three sampled shard-00-00 step-downs, leaving most elapsed time after preparation. However, the available measurements do not distinguish checkpoint work from waits sufficiently to explain the duration.

The process aborts immediately after the failing checkpoints complete, so FTDC misses updated last-completed checkpoint timings. This leaves incomplete measurements for comparison with routine checkpoints.

Proposed solution

Reproduce the behaviour using the Python workload attached to HELP-101625 and compare routine and step-down checkpoints under comparable load.

Collect targeted phase timings and work-volume statistics, supported by profiling, to identify where the additional time is spent. Capture the measurements for failing handoffs as well as successful ones.

Definition of Done

• Repeated workload runs characterize routine and step-down checkpoint durations under comparable load.
• Targeted measurements distinguish additional checkpoint work from waits and identify the phases responsible for the longer duration.
• Findings and any follow-up performance or observability work are documented.