ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction
    • None
    • Storage Engines - Transactions
    • 96.072
    • SE Transactions - 2026-12-18
    • None

      Problem

      When the bytes in cache reach the midpoint of eviction_target and eviction_trigger (85% with eviction_target=75, 87.5% with the defaults), the eviction server sets WT_CACHE_EVICT_NOKEEP. While it is set, __wt_page_in_func marks every page read from disk WT_READGEN_WONT_NEED, internal pages included.

      For an internal page this leads to evictions that cannot succeed:

      1. The reading thread tries to force-evict it on release (__wt_page_release -> __wt_page_release_evict) while it still holds the child it is descending to, so the attempt fails on active children.
      2. Later the eviction walk releases the page with WT_READ_NO_EVICT. __wt_page_evict_soon_check sees the won't-need read generation and __wt_page_evict_urgent queues it, before the walk drops its own hazard pointer. Workers take the urgent queue first and fail on active children, or on the walk's hazard pointer. __evict_page bumps the read generation before the attempt, so after the failure the page stays as an ordinary, recently used page.

      Evidence

      • HELP-103000 (8.0.34 and 8.0.37, shard 0/1 nodes, per-minute FTDC). In minutes with no-keep set at least 90% of the time, urgent pushes from the release path were 1,000-5,500/s, against 37-153/s in minutes with no-keep at most 10% of the time. Hazard-pointer blocks were 230-1,250/s against 17-201/s, and active-children failures 950-2,870/s against 220-400/s. With no no-keep at all (Nov 7-8), urgent pushes were 3-8/s. Checkpoint page visits do not correlate (r -0.16 to 0.12).
      • Local reproduction on 8.0.37 with temporary counters: 5M rows x 900 B with 4 indexes, 128 MB cache, 4 eviction threads, random point readers and a deleter walking one index in order. Every urgent push came from the eviction walk releasing a won't-need page, mostly internal pages. 73-97% of urgent attempts failed on active children, and the hazard-pointer holder on the urgent queue was the eviction walk in almost every case. Application threads that were themselves evicting held 0-3 blocking hazard pointers per minute.
      • A/B with no-keep applied to leaf pages only (two runs each):
        • Deleter only: urgent pushes 37/s -> 0, forced-eviction failures 40/s -> 0, average fill 79-80% -> 64-66%, no-keep time 25-35% -> 7-12%.
        • Readers plus deleter: urgent pushes 6.5/s -> 0.15/s, forced-eviction failures 12.7/s -> 2.5/s, fill and hazard blocks unchanged.

      In HELP-103000 this amplifies the problem once the cache is past the no-keep threshold, but it is not the root cause: the eviction workers find the queue empty on 86-96% of refills there.

      To investigate

      • Leaf-only no-keep, against a narrower option: keep internal pages won't-need, but attempt forced or urgent eviction only when the page has no resident child.
      • Side effect to check: today a won't-need internal page is reclaimed quickly once its children are gone. With leaf-only no-keep it becomes an ordinary page, and the walk only queues a childless internal page when the tree is backed off or eviction is aggressive. A large one-pass scan under no-keep could then leave many internal pages in cache.
      • Workloads: a full scan under no-keep, random reads plus a deleter, and a large tree with a high internal-page share. Metrics: internal bytes in cache, internal reads and evictions, average fill and no-keep time, application-thread eviction and cache wait.
      • The local workload reads from the OS page cache at 350-500k pages/s, against 3-14k/s on the HELP-103000 nodes, so the fill effect needs checking on a production-shaped setup.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: