test/format (disagg) disagg.stepdown_async combined with debug.slow_checkpoint can exceed the 15-minute run watchdog

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • WT12.0.0, 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Test Format
    • None
    • Storage Engines - Transactions
    • 116.593
    • SE Transactions - 2026-09-11, SE Transactions - 2026-09-25
    • 2

      Two recent duplicates auto-linked to WT-18250 (BFG-3609221 on 2026-09-01, BFG-3627928 on 2026-09-07, both on `stepdown-async` variants, both after WT-18410/WT-18458/WT-18478/WT-18211 landed) show a different signature from the original WT-18250 report: at abort time, `cache dirty check: no`, `cache updates check: no`, `cache full: no` — none of the cache pressure triggers were active. The original ticket's auto-resolution rule ORs "format run more than 15 minutes past the maximum time" with a dirty-trigger phrase, so these are being bucketed under WT-18250 even though the mechanism differs.

      Cache dump at the moment of abort (BFG-3609221) shows both ingest btrees 100% dirty with zero clean pages, while the stable trees had fully drained. The periodic WiredTigerStat log for the 5 minutes leading up to the abort shows "tracked dirty pages in the cache from the ingest btrees" perfectly flat the entire window, while "eviction server skips pages that have been reconciled previously at the same prune timestamp" (WT-18410's fix) fires 2.5M+ times — the fix is engaging correctly, and the eviction server thread is actively walking (not idle) in the core dump.

      Ingest-table dirty content is retired via checkpoint pickup advancing the ingest table's prune timestamp (__wti_layered_iterate_ingest_tables_for_gc_pruning, called from checkpoint metadata adoption), not ordinary eviction. During disagg.stepdown_async, the step-down thread runs its own checkpoint while all worker writes are paused (format_disagg.c), so no new checkpoint is picked up until that checkpoint finishes.

      debug.slow_checkpoint injects a 10ms sleep per dirty internal page written during checkpoint sync (bt_sync.c, WT_CONN_DEBUG_SLOW_CKPT). At abort time the cache dump shows roughly 44,000 dirty internal pages across the open trees, so this stress knob alone can add on the order of 7+ minutes to the step-down checkpoint. Combined with the step-down protocol's own drain (120s) and write-pause (60s) timeouts, the total can approach or exceed format's fixed 15-minute abort timer with no correctness bug involved — two self-inflicted format stress knobs compounding against the same wall clock.

      Fix: disable debug.slow_checkpoint when disagg.stepdown_async is configured, following the existing precedent in config_disagg_storage() that already disables ops.prepare/ops.truncate for the same reason (the step-down drain doesn't account for them either).

            Assignee:
            Chenhao Qu
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: