Investigate fast-truncate dirty cache growth during PIT oplog apply

XMLWordPrintableJSON

    • Storage Engines, Storage Engines - Transactions
    • 0.004
    • SE Transactions - 2026-09-25
    • 5

      Background

      HELP-99633 reported PIT restore becoming effectively stuck while applying replicated truncateRange oplog entries. The service-side mitigation in SERVER-134701 limits an apply batch to one truncateRange, allowing progress and resolving the immediate operational problem.

      A WiredTiger follow-up is still needed to understand the underlying cache behavior. During the incident's four-second FTDC transition:

      • 23 WT cursor truncate calls and 23 applied operations were recorded.
      • 529 fully covered leaf pages were fast-deleted.
      • 29 leaf pages were read into cache on the slow path.
      • local.oplog.rs dirty-leaf bytes increased by 263.79 MiB.
      • Global tracked dirty bytes increased by a net 187.00 MiB, from 238.38 MiB to 425.38 MiB, crossing the 409.6 MiB dirty trigger.

      The counters prove that fast truncate was active, but they do not establish which pages contributed the table-local dirty-leaf growth. In particular, the growth must not be attributed uniformly to the fast-deleted pages or to the 29 leaf reads without further evidence.

      Investigation goals

      • Explain why this workload produced such a large dirty-leaf increase while fast truncate was active.
      • Verify whether fast truncate eligibility and execution were working as intended for the incident's time-based oplog ranges.
      • Attribute dirty accounting among fast-delete parent/reference updates, slow-path boundary or fallback leaves, resident-page state transitions, and any disaggregated-storage update restoration.
      • Identify why leaves fell back to the slow path and whether those fallbacks can be reduced.
      • Quantify fast truncate efficiency and cache-pressure behavior under an incident-like oplog workload.
      • Determine whether WiredTiger can further reduce dirty-byte growth, eviction pressure, or truncate latency without weakening correctness.
      • Coordinate with WT-18637, which tracks a reusable fast-vs-slow truncate performance benchmark, and avoid duplicating that benchmark work.

      Definition of done

      • Reproduce or instrument an incident-like truncate workload sufficiently to explain the dirty-byte accounting.
      • Document the fast-path and slow-path page counts, fallback reasons, and dirty-byte contributions with supporting counters or tracing.
      • Confirm whether the observed behavior is expected or identifies an implementation inefficiency.
      • If an optimization is justified, describe the proposed change and add implementation and test follow-ups; otherwise document why the current behavior is necessary.
      • Record the findings on HELP-99633 and cross-reference any resulting work.

            Assignee:
            Shoufu Du
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: