-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Checkpoints
-
None
-
Storage Engines - Transactions
-
311.436
-
SE Transactions - 2026-11-06
-
1
During checkpoints, __wt_sync_file drains the global eviction generation for every tree, forcing each tree to wait for unrelated evictions across the database. This change reads btree->evict_busy after setting WT_BTREE_SYNC_WAIT and skips __wt_gen_next_drain when no evictions are running on that tree. Skipping the drain avoids these waits and reduces checkpoint sync time.
Follows WT-12541, which adds statistics for this wait (PR https://github.com/wiredtiger/wiredtiger/pull/14948).
In WT-18829 (HELP-101149, MongoDB 8.0.32 and 8.0.34), a checkpoint syncs roughly 15k trees. It costs about 2 ms per tree on the primary versus 0.7 ms on a secondary, while application threads on the primary evict about 31k pages/s. __wt_sync_file calls __wt_gen_next_drain(session, WT_GEN_EVICT) for every tree. The eviction generation covers all of __wt_evict, including history store inserts (src/evict/evict_page.c, WT_ENTER_GENERATION / WT_LEAVE_GENERATION). Every tree therefore waits for the longest eviction running anywhere in the database. FTDC cannot show how much this wait contributes to the per-tree cost.
Only evictions on the tree being synced can conflict with its sync: they read its sync state, free its blocks, and split into its internal pages. Every caller running __wt_evict on the tree increments btree->evict_busy first (__evict_get_ref, __wt_page_release_evict, __wti_delete_page). __wt_evict_file and salvage require exclusive access to the tree.
The change:
- After storing WT_BTREE_SYNC_WAIT, issue a full barrier and execute an acquire read of evict_busy. If it is zero, skip __wt_gen_next_drain and increment checkpoint_evict_drain_skipped. Otherwise, drain as before.
- Add a full barrier in __wt_evict for nested evictions, which do not enter the eviction generation and miss the barrier in __wt_session_gen_enter.
Either the checkpoint observes the active eviction and drains, or the eviction observes WAIT or RUNNING in __evict_review and takes the same restrictions as an eviction starting after the drain. Finished evictions decrement evict_busy with a sequentially consistent atomic update, synchronizing with the acquire read. Before WT-3261, checkpoint waited for this counter to reach zero via __wt_evict_file_exclusive_on. This change keeps the WT-3261 generation drain and checks the counter once before draining.
Measurements
Diagnostic build on macOS arm64 (Apple M4 Pro), shared with other builds. Workload: 6 writer threads insert 2 KB values into N tables with a 30 MB cache; one thread runs checkpoints back to back. "Always drain" is this change with the skip disabled. Values are medians of 8 alternating runs.
| Tables | Build | Run time | Checkpoints | Drains run / skipped | Drain time per checkpoint | Mean drain | Mean checkpoint time |
|---|---|---|---|---|---|---|---|
| 200 | always drain | 15 s | 4 | 900 / 0 | 675 ms | 3.4 ms | 3724 ms |
| 200 | this change | 15 s | 6 | 9 / 1085 | 11 ms | 6.7 ms | 3002 ms |
| 8 | always drain | 10 s | 18 | 166 / 0 | 118 ms | 12.4 ms | 589 ms |
| 8 | this change | 10 s | 21 | 68 / 126 | 45 ms | 14.1 ms | 486 ms |
With only the WT-12541 statistics on the same 8-table workload, statistics showed 115 drains and 2.54 s total drain time across 12 checkpoints in 10 s.
Local testing
Built with RelWithDebInfo, HAVE_DIAGNOSTIC=1, ENABLE_STRICT=1 (-Werror), Python, and Apple clang 21. dist/s_all with clang-format 23 produced no changes to files modified in these PRs.
- WT-12541 statistics PR: run.py -j 4 stat passed (Ran 158 tests). run.py -j 4 checkpoint ran 399 tests; the 8 failures were in test_checkpoint_crash01, whose child run.py requires a directory named build. With WT_BUILDDIR set, both test_checkpoint_crash01 and test_checkpoint_crash02 passed.
- This change: run.py
j 4 checkpoint eviction hs statpassed with WT_BUILDDIR set (Ran 1071 tests). test/checkpoint/test_checkpoint -t r -T 8 -W 3 -n 20000 -k 20000 -r 2 returned rc=0 unnamed and with -c mycheckpoint. csuite schema_abort smoke.sh returned rc=0. test/format with checkpoint=on, cache=20, 8 tables, 8 threads, and 2-minute runs passed 3 of 3 runs, and 3 of 3 with stress.checkpoint and stress.split_1split_8. The local workload passed 20 of 20 runs. - Mutations checked diagnostic assertions in __wti_block_alloc and __wti_block_off_free (only checkpoint may allocate or free blocks in RUNNING sync state). Skipping the drain unconditionally caused test/format to abort 6 of 6 runs, test/checkpoint to abort, the first schema_abort configuration to abort, and the workload to abort 5 of 5 runs. Skipping the drain only when eviction was active caused test/format to abort 2 of 3 runs and the workload to abort 5 of 5 runs. The memory barriers cannot be validated by mutation on this hardware.
Pull request: https://github.com/wiredtiger/wiredtiger/pull/14949