Issue Summary

When a node has a large number of data handles and its dirty content is spread thinly across many of them, the eviction server cannot queue dirty pages fast enough. Eviction throughput stays flat while dirty content climbs to the trigger, application threads are pulled into eviction, and writes stall. Adding eviction worker threads does not help because the limit is candidate supply, not the number of consumers.

Observed on a customer primary (HELP-101149, 8.0.32 and 8.0.34, 96 cores, 240 GiB cache, 12 eviction threads). The numbers below come from FTDC. The per-tree distribution of dirty content could not be measured because FTDC has no per-collection stats for application namespaces, so that part is an inference.

Context

Workload shape

• The primary has about 143k active data handles (range 112k to 213k). The secondaries have about 104k.
• Only about 15k trees are checkpointed per checkpoint cycle, roughly 10% of the active handles. Updates are spread across many trees, and none dominates.
• Dirty content is above the 5% target 71% of the time and above the 20% trigger 27% of the time (277 episodes, about 4 hours out of 15).

Eviction behavior on the primary

• Eviction throughput is about 36k to 40k pages per second in every dirty-content state. It does not rise as pressure rises.
• Candidate supply is also flat at about 40k to 43k pages per second in every state.
• Above the trigger, application threads take over about 31k evictions per second from the workers (worker rate drops from about 35k to about 9k per second). The total does not change.
• Above the trigger, the candidate queue is empty on about 89% of refills and on about 61% of requests for a page. Below the trigger it is rarely empty.
• The server refills the queue about 450 to 650 times per second. Each refill adds at most 100 entries, and above the trigger refills are nearly full (about 98 entries each). Supply is therefore bounded by refill rate times the per-refill cap.
• Each refill visits about 15 to 28 trees. Each tree visit sees about 10 pages and queues about 3. About 15% to 17% of tree visits give up with no candidates. About two thirds of visits target fewer than 10 pages and fewer than 0.2% target 32 or more.
• The data handle count itself is not the cost. Within the same run the handle count varied by almost 2x. Handle visits per refill doubled while the refill rate stayed flat and supply fell only 7% to 18%. The cost is in visiting many trees that each yield very little.
• A secondary with fewer handles has similar supply but far lower demand, so it never reaches the trigger.

Impact

• Write latency averages about 3 s on the primary with peaks near 27 s. Queued writers reach about 7k.
• Checkpoints run about 67% of the time on the primary. Duration tracks the dirty level at checkpoint start. Per-tree checkpoint cost is about 2 ms per tree on the primary against 0.7 ms on a secondary.
• Replication lag and flow control engage after each stall.

Related

• WT-18120 (walk starvation, 8.0) lets the server walk trees that dominate the cache despite their skip period. That covers a few large trees, not this many-small-trees case, and it is not in the customer's version.

Goal

Improve eviction server performance by 20% to 30% on the reproducer. The target is not a fundamental fix for the candidate supply limit. It is to buy time until one is found. All options below are ways to reach this target.

Proposed Solution

Reproduce first, then pick approaches based on what the profile shows. Please create a sub-ticket for each optimization pursued, with its own measurement against the reproducer, so each can be reviewed, backported or dropped on its own.

Reproduction

• Create 100k or more collections and indexes, apply uniformly spread updates so about 10% of trees are dirty at any time, and size the dirty target and trigger so the workload reaches them.
• Record candidate supply, refill rate, tree visits per refill and yield per tree visit as the number of trees varies.

Option 1: Raise the per-refill cap or refill more often
Pros:

• Small change and easy to evaluate.
  Cons:
• A larger queue holds candidates for longer, which may make older entries stale.
• Does not fix the low yield per tree visit.

Option 2: Prefer trees with more dirty content when choosing where to walk
Pros:

• Each tree visit yields more pages for the same fixed cost.
  Cons:
• Needs a cheap per-tree dirty estimate and must avoid starving small trees.

Option 3: Reduce the cost of visiting a tree that yields few candidates
Pros:

• Helps even when dirty content cannot be concentrated.
  Cons:
• Needs profiling to find out where the time goes. FTDC has no per-thread CPU.

Option 4: Tactical optimizations of the eviction server
Small, independent changes to the existing server:

• Skip more data handles that cannot yield target pages, so each refill spends its time on trees that can contribute. Candidates to evaluate: trees with no dirty content when the server is evicting for dirty pressure, trees that recently gave up with no candidates (back off for a period), and trees too small to contribute.
• Pin the eviction server thread to a dedicated CPU when it is struggling to find enough pages (for example, when the candidate queue is empty on most refills). This avoids scheduler delays and migrations on a busy host. Run it only when the pressure condition holds and make it configurable.
• Further items found while profiling the server (see Option 3).
  Pros:
• Low-risk changes that can ship and be backported independently.
• Gains add up, and none blocks a fundamental fix.
  Cons:
• Does not remove the cap on candidate supply.
• CPU pinning needs a spare core, depends on the platform and container CPU limits, and may hurt if the pinned core is contended.

Out of scope: long-term solution
Walking trees in parallel with more than one eviction server may be the long-term solution. It raises the ceiling directly, but it is a larger change with new contention on the walk lock, too much work for an optimization, so it is not part of this ticket.

Definition of Done

• A reproducer shows candidate supply falling, or application threads taking over eviction, as the number of dirty trees grows.
• The combined changes raise eviction server performance by 20% to 30% on that reproducer, without hurting a workload with few large trees.
• Candidate supply, refill rate and yield per tree visit are visible in statistics so the next occurrence can be diagnosed without a reproducer.