Oplog truncation re-walks every unreclaimed deleted page because each pass starts from a null lower bound

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      Every size-based oplog truncation pass calls the range truncate with a null minimum RecordId (oplog_truncation.cpp performUnreplicatedTruncate, and performReplicatedTruncate in the disagg replicated truncation thread). With no start key, WiredTiger positions the start cursor with a bare next() from the beginning of the table (_wt_session_range_truncate), which walks over every deleted-but-not-yet-reclaimed page ref left by all previous passes before reaching live data.

      Those refs are only removed when checkpoint cleanup reclaims them, which is gated on the truncation becoming globally visible. Until then the walk cost compounds: each pass pays O(backlog) ref-skip checks, and the same dead ground is re-walked on every pass.

      Evidence

      Sys-perf reproduction (disagg 2-node, 990MB oplog, 5 minute retention, truncation points forced to 1MB so the per-pass overhead dominates): the maintainer thread saturates while per-pass cost tracks the per-table FTDC counter cursor: Total number of deleted leaf/internal pages skipped during tree walk linearly at roughly 2.5us per skipped ref (13.9k skips -> 31ms per pass, 70.4k skips -> 186ms per pass). Read volume and WT delete work per pass stay constant, so the walk is the entire variable cost. During a 40 minute window where checkpoint cleanup reclaimed nothing, truncation fell behind ingest and the oplog overshot its cap 15x.

      Patch runs: stock behavior https://spruce.corp.mongodb.com/version/6a9ec14ba924730007818e26, CPU profile https://spruce.corp.mongodb.com/version/6a9f4359c60f1700074e4135.

      The same mechanism is visible at default 16MB truncation points as per-pass cost variance (33ms to 166ms), since the backlog between checkpoint reclaims is workload dependent.

      Proposed fix

      Thread the previous marker's lastRecord through TruncateFn (truncateByMarkerQueue already tracks it as highestTruncated) and pass it as the truncation lower bound. With a start position WiredTiger positions by key (a bounded cursor bound() + next(), or equivalently a start key resolved via search_near) - a root-to-leaf descent - and the dead zone, which lies entirely below the bound, is never visited. The first pass of each reclaim keeps the null bound, so behavior there is unchanged.

      Both the unreplicated path and the disagg replicated path need the bound.

            Assignee:
            Unassigned
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: