ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction
    • None
    • Storage Engines
    • 133.227
    • None
    • None

      Objective

      Reproduce the large-cache, update-trigger-only failure mode in standalone WiredTiger: update pressure reaches its trigger, but eviction does not drain update bytes effectively, leaving application threads waiting. This is a symptom-reproduction task; the underlying cause is not established.

      Target signature

      • Large cache: approximately 77 GiB.
      • Total cache occupancy remains around 75–85%; dirty content remains at or below 3%; update bytes rise toward the 20% update trigger (accept 18–22%).
      • The update trigger advances and the updates-hard state is present; dirty and full-cache triggers do not advance during the same interval.
      • Empty score remains at least 95, cache-wait time advances, and application write latency is elevated.
      • Require the signature for a 60-second detection window and a following non-overlapping 60-second confirmation window. Do not count windows across process/configuration changes, missing samples, counter resets, or gaps.

      Customer reference at peak: updates 20.0%, dirty 1.70%, total occupancy 79.66%, empty score 99–100, and update-hard eviction state. The candidate queue was empty for about 98% of top-ups; roughly 275k pages/s were seen versus 3k queued/s, with about 32 unmodified-page evictions/s. These are mechanism fingerprints to investigate, not independent pass criteria.

      Current evidence and investigation

      Prior standalone WT 8.3.2 runs did not reproduce the signature: update bytes stayed around 2.3–2.5%, the update trigger did not fire, and cache wait did not increase. The customer workload is primarily in-place modifies plus substantial index insert/remove churn (about 460 document modifies/s within about 1.2k transactions/s), retryable-write activity on a small hot transactions table, and an approximately 300-second oldest-timestamp window. Customer FTDC shows a large internal-page share (about 29%) and poor candidate-walk yield. Tree size, internal-page residency, and candidate-supply limitations remain hypotheses to test, not confirmed causes.

      Use the existing Alibaba reproduction investigation as background. Preserve the distinction between customer-measured values, synthetic workload assumptions, and locally reproduced results. Do not claim an exact root cause from matching the symptom alone.

      Definition of done

      1. Produce a repeatable standalone WT workload/configuration and document the exact WT revision, effective settings, workload, and run procedure.
      2. Demonstrate the update-only trigger signature and cache-wait/latency impact over both required windows, with dirty/full triggers remaining inactive.
      3. Retain bounded measurements sufficient to show update-byte trajectory, eviction candidate supply and outcomes, and relevant workload rates; explain any remaining difference from the customer signature.
      4. If the signature cannot be reproduced, document the tested conditions and the strongest remaining gap/hypothesis rather than reporting a match.

      Implementation of a long-term eviction redesign is out of scope for this task.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: