Dhandle scaling Perf: Keep a list of modified dhandles for checkpoint prepare

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: DHandles
    • None

      This comes out of the investigation in WT-17617. There is not yet a POC for this.

      Track every dhandle that has been modified since the last checkpoint, then we don't have to scan the entire dhandle list.  Basically we keep a list of dhandles.  Whenever an operations modifies a btree, at the point at which it marks the btree as 'modified' (probably currently an atomic operation) - now the operation would need to be a CAS.  If the operation is the CAS "winner", it should move the btree's dhandle to the modified list.  When a btree is checkpointed, it should be removed from the modified list.

      Currently, determine what dhandles are in the checkpoint involves walking the entire dhandle list (which can be 100k or greater entries) and examining each btree.  However the number of modified dhandles in a checkpoint should typically be small (and smaller for infinity, with 5 second checkpoints).  So we basically already have the list when we're starting.

      The hard part about this ticket is reasoning about the edge cases.  Specifically, when adding to list races with checkpointing beginning and ending handle gathering. There's probably lockfree ways to do this, but it may not be needed.  We could probably afford having a locked modify handle list.  The only time an addition is done is on the very first btree modification.

      Part of the work here is also to measure before and after, and potentially add needed stats.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Donald Anderson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: