ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Checkpoints
    • None
    • Storage Engines - Transactions
    • 311.436
    • SE Transactions - 2026-11-06
    • 1

      During checkpoints, __wt_sync_file drains the global eviction generation for every tree, forcing each tree to wait for unrelated evictions across the database. This change reads btree->evict_busy after setting WT_BTREE_SYNC_WAIT and skips __wt_gen_next_drain when no evictions are running on that tree. Skipping the drain avoids these waits and reduces checkpoint sync time.

      Follows WT-12541, which adds statistics for this wait (PR https://github.com/wiredtiger/wiredtiger/pull/14948).

      In WT-18829 (HELP-101149, MongoDB 8.0.32 and 8.0.34), a checkpoint syncs roughly 15k trees. It costs about 2 ms per tree on the primary versus 0.7 ms on a secondary, while application threads on the primary evict about 31k pages/s. __wt_sync_file calls __wt_gen_next_drain(session, WT_GEN_EVICT) for every tree. The eviction generation covers all of __wt_evict, including history store inserts (src/evict/evict_page.c, WT_ENTER_GENERATION / WT_LEAVE_GENERATION). Every tree therefore waits for the longest eviction running anywhere in the database. FTDC cannot show how much this wait contributes to the per-tree cost.

      Only evictions on the tree being synced can conflict with its sync: they read its sync state, free its blocks, and split into its internal pages. Every caller running __wt_evict on the tree increments btree->evict_busy first (__evict_get_ref, __wt_page_release_evict, __wti_delete_page). __wt_evict_file and salvage require exclusive access to the tree.

      The change:

      • After storing WT_BTREE_SYNC_WAIT, issue a full barrier and execute an acquire read of evict_busy. If it is zero, skip __wt_gen_next_drain and increment checkpoint_evict_drain_skipped. Otherwise, drain as before.
      • Add a full barrier in __wt_evict for nested evictions, which do not enter the eviction generation and miss the barrier in __wt_session_gen_enter.

      Either the checkpoint observes the active eviction and drains, or the eviction observes WAIT or RUNNING in __evict_review and takes the same restrictions as an eviction starting after the drain. Finished evictions decrement evict_busy with a sequentially consistent atomic update, synchronizing with the acquire read. Before WT-3261, checkpoint waited for this counter to reach zero via __wt_evict_file_exclusive_on. This change keeps the WT-3261 generation drain and checks the counter once before draining.

      Measurements

      Diagnostic build on macOS arm64 (Apple M4 Pro), shared with other builds. Workload: 6 writer threads insert 2 KB values into N tables with a 30 MB cache; one thread runs checkpoints back to back. "Always drain" is this change with the skip disabled. Values are medians of 8 alternating runs.

      Tables Build Run time Checkpoints Drains run / skipped Drain time per checkpoint Mean drain Mean checkpoint time
      200 always drain 15 s 4 900 / 0 675 ms 3.4 ms 3724 ms
      200 this change 15 s 6 9 / 1085 11 ms 6.7 ms 3002 ms
      8 always drain 10 s 18 166 / 0 118 ms 12.4 ms 589 ms
      8 this change 10 s 21 68 / 126 45 ms 14.1 ms 486 ms

      With only the WT-12541 statistics on the same 8-table workload, statistics showed 115 drains and 2.54 s total drain time across 12 checkpoints in 10 s.

      Local testing

      Built with RelWithDebInfo, HAVE_DIAGNOSTIC=1, ENABLE_STRICT=1 (-Werror), Python, and Apple clang 21. dist/s_all with clang-format 23 produced no changes to files modified in these PRs.

      • WT-12541 statistics PR: run.py -j 4 stat passed (Ran 158 tests). run.py -j 4 checkpoint ran 399 tests; the 8 failures were in test_checkpoint_crash01, whose child run.py requires a directory named build. With WT_BUILDDIR set, both test_checkpoint_crash01 and test_checkpoint_crash02 passed.
      • This change: run.py j 4 checkpoint eviction hs stat passed with WT_BUILDDIR set (Ran 1071 tests). test/checkpoint/test_checkpoint -t r -T 8 -W 3 -n 20000 -k 20000 -r 2 returned rc=0 unnamed and with -c mycheckpoint. csuite schema_abort smoke.sh returned rc=0. test/format with checkpoint=on, cache=20, 8 tables, 8 threads, and 2-minute runs passed 3 of 3 runs, and 3 of 3 with stress.checkpoint and stress.split_1split_8. The local workload passed 20 of 20 runs.
      • Mutations checked diagnostic assertions in __wti_block_alloc and __wti_block_off_free (only checkpoint may allocate or free blocks in RUNNING sync state). Skipping the drain unconditionally caused test/format to abort 6 of 6 runs, test/checkpoint to abort, the first schema_abort configuration to abort, and the workload to abort 5 of 5 runs. Skipping the drain only when eviction was active caused test/format to abort 2 of 3 runs and the workload to abort 5 of 5 runs. The memory barriers cannot be validated by mutation on this hardware.

      Pull request: https://github.com/wiredtiger/wiredtiger/pull/14949

            Assignee:
            Unassigned
            Reporter:
            Mikhail Sotnikov (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: