Investigate the trade-off of letting eviction reconcile ahead of a running checkpoint in disaggregated storage

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Engines - Transactions
    • 260.555
    • SE Transactions - 2026-10-09
    • 5

      Background

      Under precise checkpoint the eviction server passes over any dirty page whose newest commit sits above the pinned stable timestamp, because that content cannot be written (eviction_server_skip_pages_checkpoint_timestamp). WT-17681 made eviction reconcile under the checkpoint's published snapshot and let the checkpoint adopt the resulting image, so during the window where a checkpoint has published its snapshot and has not yet reached a tree, reconciling such a page produces exactly the image the checkpoint would write.

      A draft change relaxes that skip for precisely that window, scoped to disaggregated btrees:

      • gate on the checkpoint having published its snapshot and btree->checkpoint_gen being behind the connection checkpoint generation
      • skip a page already reconciled at the current checkpoint timestamp, so its image is only produced once per checkpoint
      • bypass the modification-count heuristic, since accumulating more changes is not a reason to wait when the checkpoint would write the page regardless

      Two statistics were added: eviction_server_pages_queued_ahead_of_checkpoint and eviction_server_skip_pages_checkpoint_timestamp_not_move.

      The draft lives on the local branch wt-18400-evict-ahead-of-checkpoint (commit 45a2f64b02), together with test_disagg_evict_ahead_of_checkpoint.py, which fails without the change and passed 20/20 with it. Not yet pushed.

      Hypothesised benefit

      Faster checkpoints let checkpoints run more frequently. More frequent checkpoints mean less data pinned in a standby's cache, which should mean a faster step-up. This is the motivating argument and it is the one thing the work has not measured.

      What was measured

      Workload linkbench2.2024-05 on disagg-m8g-perf-11-node.arm.aws (Disagg m8g 2-Node ReplSet ARM, Mongotune SLS perf, no overload protection). One patched run against mainline 2f94f37.

      The mechanism works, comprehensively:

      counter baseline patched
      eviction server skips pages > checkpoint timestamp 40,005,040 900,469
      eviction server pages queued ahead of checkpoint
      951,857
      eviction server skips: already done at this checkpoint timestamp
      9,400,668
      checkpoint pages whose reconciliation was skipped (adopted) 2,259 761,164

      Normalised per user update (both runs ran an identical 601 s request phase at equal throughput, so this is per unit of user work):

      ratio baseline patched delta
      checkpoint reconciliation 2.032 1.733 -14.7%
      eviction reconciliation 0.993 1.882 +89%
      disaggregated block manager put 3.062 3.116 +1.8%

      77% of the extra eviction reconciliation was adopted by the checkpoint (extra 0.888/update, adopted 0.683/update).

      Checkpoint side, single run, unnormalised:

      • average checkpoint duration 12.85 s -> 12.25 s (-4.6%)
      • bytes reconciled 61.83 GB -> 43.38 GB, history store pages reconciled 33,278 -> 1,508
      • leaf pages visited per checkpoint essentially unchanged, 10,336,575 -> 10,545,861

      Costs beyond the extra reconciliation:

      • page eviction blocked in disagg, "can only be written by the next checkpoint": 91,186 -> 160,490
      • eviction server no-progress sleeps: 102,140 -> 211,100

      Reported performance: neutral

      Against the correct baseline 2f94f37:

      measurement baseline patched delta
      REQUEST_PHASE_THROUGHPUT_OVERALL 38,690.0 38,716.6 +0.1%
      LOAD_PHASE_THROUGHPUT_OVERALL 329,661.5 332,431.8 +0.8%
      p99 latencies (all operations)     -0.8% to -10%
      p95 latencies (all operations)     -4.6% to +1.4%

      CV of the stable region is 4.36%, so none of this is outside noise. There is no throughput gain and no tail regression.

      Note for anyone re-running this: an earlier comparison used mainline ddb1615 (2026-08-21) and appeared to show +6.3% throughput with 10-35% p99 regressions. That was entirely an artefact of the stale baseline - mainline itself moved between ddb1615 and 2f94f37 (throughput +6.2%, ADD_LINK p99 15.3 -> 18.8). Compare against the immediately preceding mainline commit, and prefer the chart's stable-region mean and CV over differencing individual runs.

      Questions to settle

      1. Does a shorter checkpoint actually reduce what a standby pins, and does that speed up step-up? Unmeasured, and it is the whole justification. linkbench2 does not run checkpoints at a rate where this would show, so this likely needs a purpose-built test with a shortened checkpoint interval, measuring follower cache footprint and step-up time.
      2. Is the -4.6% checkpoint duration real? Single run, and with a 4.36% CV on throughput the variance on checkpoint duration is probably comparable. Needs repetitions.
      3. Why did adopting 761,164 reconciliations reduce checkpoint reconciliation by so much less than that? Per update the checkpoint saved 0.299 while adopting 0.683, so roughly 44% of adoptions converted into avoided work. Hypothesis, unconfirmed: update-restore eviction does an in-memory split and __split_insert clears rec_ckpt_snap_gen, so the pre-reconciled image is discarded and the checkpoint reconciles the result anyway. Check whether the in-memory split counters rise in step with the extra evictions.
      4. Does the waste scale with checkpoint frequency? The not_move guard is keyed to the current checkpoint timestamp, so it resets every checkpoint and a hot page may be pre-reconciled once per checkpoint. If frequency is the goal, this gets worse in exactly the regime we want. Consider gating on how much the page changed since its last pre-reconciliation rather than on the timestamp having moved.
      5. Can the queue-to-visit race be reduced? The walk queues a page from an unvisited tree and the checkpoint reaches that tree before a worker dequeues it, giving 91,186 -> 160,490 rejections and doubled no-progress sleeps. Gating on trees the checkpoint will not reach soon, rather than merely has not reached yet, would cut this.

      Where it does not help

      Screening baseline FTDC across the disagg workloads showed eviction server unable to reach eviction goal is 0 in every phase measured, and no cache stuck metric exists there. There is no cache-stuck pressure in the disagg perf fleet for this change to relieve, so cache stuck is not a supporting argument.

      data_handle_locust (20,000 collections) is the wrong workload for this: eviction there is dominated by the materialization frontier (31,992 blocks) and history store skips (22,074), with the checkpoint-timestamp skip only third at 52,437. The relaxation fired 3,984 times and changed nothing.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: