Improve per-marker overhead of oplog truncation logic

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Replication
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Investigate oplog truncation logic that incurs a fixed per-marker overhead. This can become a performance issue in cases where a large marker queue is present. In some cases the truncation thread is truncating in a continuous loop, as seen in some large-scale loads such as HELP-99461 so any logic we run per-marker contributes directly to truncation latency.

      Generally right now we re-evaluate whether each truncation marker can be truncated by recomputing the truncation bounds. If we can assume that the truncation point never goes backwards, we should be able to cache it and only recompute when the next marker that we're checking is behind it.

      Findings from claude:

      1.  O(n²) drain loop. _hasExcessMarkers sums marker.bytes over the entire deque every call (oplog_truncate_markers.cpp:316-320), called once per truncate iteration. Draining N markers walks the deque N times under _markersMutex. Fix: maintain a running total on push/pop.
      2. Redundant getPinnedOplog(). Computed once per drain pass (oplog_truncation.cpp:206) then recomputed inside _hasExcessMarkers on every iteration (oplog_truncate_markers.cpp:330-331), which also takes _oplogPinnedByBackupMutex.
      3. _readEarliestTimestamp on every truncate (wiredtiger_record_store.cpp:1527) — an extra cursor open+read solely to refresh a cache; could be derived from the truncate bound instead of re-fetched.
      4. checkOplogTruncationBounds re-derives the first record and next-after-marker record every iteration (oplog_truncation.cpp:48,76-77), even though the prior iteration already truncated up to that exact point.
      5. No batching of adjacent markers into one truncateRange entry. Each marker gets its own WUOW, oplog-entry write, and oplog-slot reservation (replicated_oplog_truncation_thread.cpp:63-70). Markers are contiguous by construction and the entry already carries an explicit range + aggregate counts, so N adjacent markers could collapse into 1 truncate call — this is a lever independent of #1 (helps even if marker size itself doesn't change).
      6. Self-feeding accounting. The truncate's own replicated oplog entry runs updateCurrentMarkerAfterInsertOnCommit (oplog.cpp:288-311), so each truncate adds bytes back into the marker queue it's draining. Minor, but worth closing.
      7. Redundant re-entry in the drain loop. _deleteExcessDocuments releases/reacquires the global lock and calls awaitHasExcessMarkersOrDead again on success (oplog_cap_maintainer_thread.cpp:212-230,436-438), repeating the O scan from #2 each time.

        Worth flagging, not clearly a bug: the truncate's WUOW being open blocks OplogProvider from shipping past that timestamp (oplog visibility hole, oplog_provider.cpp:618-621) — ties truncate commit latency directly to replication-shipping progress. Not itself fixable cheaply, but it means fixes #2-#6 (which shrink per-truncate latency) also reduce this stall's exposure.

            Assignee:
            Unassigned
            Reporter:
            Anna Veselova
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: