• Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction
    • None
    • Storage Engines
    • 4.368
    • None
    • None

      Symptom
      A tree that holds most of the cache and still has clean leaves to evict can be held at the maximum walk period (100). The server then visits it once in about 100 handle-list rounds, the eviction queue stays empty, and the cache sits at the midpoint between eviction_target and eviction_trigger, where application threads evict the pages they have just read. It lasts until one visit happens to start near evictable leaves.

      How it gets there (r8.0.37; the same test is on develop):

      1. When the fill is back at the target the server stops and clears every saved walk position, so the tree's next walk is a fresh start.
      2. A fresh start that lands where nothing can be queued (internal pages with a resident child, dirty leaves in a clean-only walk) queues less than half of its target and the walk period doubles. Eight such walks take it from 0 to 100.
      3. At 100 the dominating-tree override of WT-18120 does not apply:
        if (evict_walk_period != 0 && btree->evict_walk_skips++ < evict_walk_period) {
            if (evict_walk_period >= WT_EVICT_WALK_PERIOD_MAX ||
              !__wt_btree_dominating_cache(session, btree, dominating_flags)) {
                ...
                continue;
        
      4. Each of the rare visits is again a fresh start, so the state ends only when one lands on evictable leaves.

      Repro (r8.0.37 with instrumentation, WiredTiger API, 88 MB cache)
      One large table holding 58-73% of the cache; point reads on a hot range 1.27 times what fits; 30 cold point reads per second across the rest of the tree; clean-only walks; 4 eviction threads; 900 s of load. Nothing is forced.

        r8.0.37 Override also applied at period 100 (experiment)
      Time with the cache at the midpoint 74% 0.2%
      Handle-list rounds in which the tree was skipped 96.6% 0%
      Pages evicted by the thread that read them, per second 3,775 7
      Server walk passes per second, all trees 8,917 879

      The second column is an experiment to confirm the cause, not a proposed change. Removing the exclusion removed the state in six runs of six (one of them together with WT-19030). WT-19030 alone did not change it (70% of the time at the midpoint).

      Limits: the repro tree is small and synthetic (1 KB leaves, 512 B internal pages). A run with the customer's page configuration and a 40 GB cache is in progress.

      Field data (HELP-103000, 8.0.34)
      On an affected primary the cache sat at 85.4-86.6% in the incident windows, around the midpoint in force (87.5% with the default target, 85% once eviction_target was 75), with 1,026-5,834 forced-eviction selections per second and the queue empty on 86-96% of refills. The affected nodes were in no-keep mode 41-91% of the time. Per-tree statistics under load are not available, so that the collection was in this state is inferred, not measured.

      Expected
      A tree that dominates the cache and has evictable pages should not be starved of walks because earlier walks started in the wrong place. Options to evaluate, each with its walk cost: let the override apply at the maximum period; count only leaf pages toward the give-up budget of a clean walk, with a cap (the same threshold as WT-19042). No change is proposed until the larger run is in.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: