Implement garbage collection checkpoint pick-up with fast truncate design

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Truncate
    • None
    • Storage Engines - Foundations
    • None
    • None

      Problem
      The ASC Fast Truncate implementation does not work on standby nodes because it needs to write fast truncate information directly to the standby table which violates the core principle in which any update/insert operations can’t be done in standby mode. A new standby fast truncate design will be adopted to enable fast truncate.

      The new fast truncate design will live in-between the ingest and stable table. The read/write path way will be ingest -> truncate -> stable. The new fast truncate design will continuously insert truncate entries. A long truncate linked list will affect the read/write performance cost, as cursor reads/writes need to loop through more entries.

      When the follower mode picks up a new checkpoint, we can prune/delete old truncate entries that would already exist on the stable table (as part of the new checkpoint). Doing this will keep the truncate list entries low, effectively increasing read/write performance.

      What needs to be done
      When the follower mode picks up a new checkpoint and update the prune timestamp of the ingest table we can perform garbage collection. Investigate what would be most optimal place to prune old truncate entries. A functional test should validate this behaviour.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: