-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Cache and Eviction
-
None
-
Storage Engines
-
4.368
-
None
-
None
Symptom
A tree that holds most of the cache and still has clean leaves to evict can be held at the maximum walk period (100). The server then visits it once in about 100 handle-list rounds, the eviction queue stays empty, and the cache sits at the midpoint between eviction_target and eviction_trigger, where application threads evict the pages they have just read. It lasts until one visit happens to start near evictable leaves.
How it gets there (r8.0.37; the same test is on develop):
- When the fill is back at the target the server stops and clears every saved walk position, so the tree's next walk is a fresh start.
- A fresh start that lands where nothing can be queued (internal pages with a resident child, dirty leaves in a clean-only walk) queues less than half of its target and the walk period doubles. Eight such walks take it from 0 to 100.
- At 100 the dominating-tree override of
WT-18120does not apply:if (evict_walk_period != 0 && btree->evict_walk_skips++ < evict_walk_period) { if (evict_walk_period >= WT_EVICT_WALK_PERIOD_MAX || !__wt_btree_dominating_cache(session, btree, dominating_flags)) { ... continue; - Each of the rare visits is again a fresh start, so the state ends only when one lands on evictable leaves.
Repro (r8.0.37 with instrumentation, WiredTiger API, 88 MB cache)
One large table holding 58-73% of the cache; point reads on a hot range 1.27 times what fits; 30 cold point reads per second across the rest of the tree; clean-only walks; 4 eviction threads; 900 s of load. Nothing is forced.
| r8.0.37 | Override also applied at period 100 (experiment) | |
|---|---|---|
| Time with the cache at the midpoint | 74% | 0.2% |
| Handle-list rounds in which the tree was skipped | 96.6% | 0% |
| Pages evicted by the thread that read them, per second | 3,775 | 7 |
| Server walk passes per second, all trees | 8,917 | 879 |
The second column is an experiment to confirm the cause, not a proposed change. Removing the exclusion removed the state in six runs of six (one of them together with WT-19030). WT-19030 alone did not change it (70% of the time at the midpoint).
Limits: the repro tree is small and synthetic (1 KB leaves, 512 B internal pages). A run with the customer's page configuration and a 40 GB cache is in progress.
Field data (HELP-103000, 8.0.34)
On an affected primary the cache sat at 85.4-86.6% in the incident windows, around the midpoint in force (87.5% with the default target, 85% once eviction_target was 75), with 1,026-5,834 forced-eviction selections per second and the queue empty on 86-96% of refills. The affected nodes were in no-keep mode 41-91% of the time. Per-tree statistics under load are not available, so that the collection was in this state is inferred, not measured.
Expected
A tree that dominates the cache and has evictable pages should not be starved of walks because earlier walks started in the wrong place. Options to evaluate, each with its walk cost: let the override apply at the maximum period; count only leaf pages toward the give-up budget of a clean walk, with a cap (the same threshold as WT-19042). No change is proposed until the larger run is in.
- is related to
-
WT-19030 Fix eviction walk backoff for a one-page remaining target
-
- Closed
-
-
WT-19042 Scale clean eviction walk budgets based on cache share
-
- In Code Review
-
-
WT-18120 Prevent eviction walk starvation for a tree dominating cache usage
-
- Closed
-
- related to
-
WT-19030 Fix eviction walk backoff for a one-page remaining target
-
- Closed
-
-
WT-19002 Reproduce the large-cache update-trigger failure mode where update bytes cannot be evicted effectively
-
- Investigating
-
-
WT-19034 Avoid small carried-over walk targets on a backed-off tree that dominates the cache
-
- Open
-
-
WT-19042 Scale clean eviction walk budgets based on cache share
-
- In Code Review
-
-
WT-18120 Prevent eviction walk starvation for a tree dominating cache usage
-
- Closed
-