Summary

Most of a checkpoint's wall-clock time is the checkpoint thread waiting for its reconciliation workers, not doing work. The sync loop drains the whole work queue every time the walk reaches an internal page, so outstanding work is capped at roughly one internal page's worth of children. On linkbench2.2024-05 over disaggregated storage that is 10.8 pages per drain and 10,004 drains per checkpoint, and the workers reach an average concurrency of 1.55 however many are configured.

Writing the leaves in one pass and the internal pages in a second cuts average checkpoint duration by 21.5%.

Where the time goes

Phase attribution was added for this investigation and validated: the per-state accumulators sum to 99% of checkpoint wall-clock time.

Splitting the sync phase:

The walk is 3.7%. Reconciliation volume is not the constraint either: an earlier experiment removing about 8% of the checkpoint's reconciliation work changed duration by less than the measurement could resolve.

Why the wait is so large

The workers are idle, not saturated. Effective concurrency is worker reconcile time divided by driver wait time: 493.1s / 317.6s = 1.55x, independent of the configured worker count, with 2065s of aggregate worker idle against 493s of reconciliation. The batch handed out between drains is 10.8 pages, about the tree's fanout, too little to spread across a pool that must then re-synchronise.

The change

Reconciling a leaf writes a new address into its parent, so an internal page cannot be written until the leaves beneath it are. That ordering does not require a drain per internal page: write every leaf first, drain once, then walk again for the internal pages. The walk returns a page beneath its children, so writing them in the order returned keeps a parent behind its descendants, and a page's dirty state is final by the time the second walk reaches it.

Each pass asks the walk for only the pages it wants. The first sets WT_READ_SKIP_INTL; the second passes a callback that skips leaves. The walk consults that callback before bringing a page in, so unwanted pages cost nothing.

The queue is drained on depth instead of on tree structure, so each queued page still holds a hazard pointer and a checkpoint still bounds what it pins in cache.

Result

Measured on linkbench2.2024-05, variant disagg-m8g-perf-11-node.arm.aws, against the same build carrying the instrumentation but not the change.

Against a ten-run mainline baseline of 13127 +/- 306ms (CV 2.19%) the duration result is roughly nine standard deviations.

Throughput is unchanged: REQUEST_PHASE_THROUGHPUT_OVERALL 36782.5 against 37025.7 for the instrumented control, a 0.7% difference against a measured 5.4% run-to-run noise floor. LOAD_PHASE is 332292 against mainline 329661.

The barriers do not fall to one per tree: they are now set by the queue-depth bound rather than by fanout.

State and what is left

Branch wt-checkpoint-two-phase-sync, unpushed. The functional change is confined to src/btree/bt_sync.c; the rest of the branch is the phase instrumentation that located the problem.

• n=1 per side on the perf comparison. The duration result holds at nine standard deviations, but "no throughput cost" needs repeats to be a claim rather than an absence of evidence.
• The instrumentation costs about 4% throughput on its own. The hot-path per-push and per-pop timers have already been removed. checkpoint_parallel_worker_idle_usecs is not scoped to a running checkpoint and should be fixed or dropped before this ships.
• format on CONFIG.disagg is 6/6 clean, against 50-75% failure rates for the earlier broken attempts. The checkpoint Python suite has not been run to completion and verify has not been exercised directly.

Rejected approach

Reconciling internal pages on the workers as well was attempted and abandoned; it is preserved on branch wt-checkpoint-parallel-intl-attempt. It is worth roughly a further 10% (internal-page reconciliation is 1.27s of a 13.2s checkpoint) but every variant decided "has this child been written" from page state, which cannot express "queued but not yet processed". Doing it correctly needs explicit queue-membership state and should be its own change gated on a format repro.

Two assumptions made during that work turned out to be false and are worth recording:

• Eviction does run on a tree being checkpointed. The guard is on modified pages only, so clean pages are discarded throughout a checkpoint, and any reference held across a checkpoint phase needs a hazard pointer.
• A page that is clean when the walk passes it may not stay clean, because writing its children dirties it. Filtering on dirty state at collection time leaves pages unwritten.