(Follower mode) Implement garbage collection checkpoint pick-up with fast truncate design

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Truncate
    • None
    • Storage Engines - Foundations
    • 1,703.315
    • SE Foundations - 2026-05-08, SE Foundations - 2026-05-22
    • 5

      Problem
      In the new standby fast truncate design, the truncate layer between ingest and stable tracks truncate ranges in an in‑memory linked list on the layered table dhandle, so the read/write path becomes ingest → truncate → stable. Each fast truncate operation appends new entries to this truncate list, and over time the list can grow long as more truncates are issued.

      A long truncate list increases the cost of both reads and writes, because cursor operations and conflict/visibility checks must scan more truncate entries to decide whether a key is visible or truncated. When a follower adopts a new checkpoint, the stable table already includes all history up to the checkpoint timestamp, so older truncate entries that fall entirely before that point can be safely discarded. The design calls for using checkpoint pick‑up to garbage‑collect obsolete truncate entries and keep the list small, but this garbage collection is not yet implemented.

      Without garbage collection on checkpoint pick‑up, followers will accumulate an ever‑growing truncate list, degrading read and write performance and increasing the cost of visibility and conflict checks, even though many entries are no longer needed once the stable table has them.

      This ticket closes that gap by implementing and validating truncate‑list garbage collection tied to checkpoint pick‑up: when follower mode adopts a new checkpoint and updates the ingest table’s prune timestamp, the system should prune/delete truncate entries that are now fully represented on the stable table, and a functional test should verify that obsolete entries are removed while active/visible truncates remain intact.

            Assignee:
            Krishen Chovhan
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: