Dhandle scaling Perf: On standby, do selective pickup of individual btree checkpoints

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Won't Do
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Checkpoints
    • None

      This comes out of the investigation in WT-17617, where this idea was POC'd.  In the data_handle workload, there are a lot of stable btrees updated in each checkpoint.  Rather than picking up all of them at once, we can choose which ones we want.  For example, for a given checkpoint, suppose that 1/Nth of the btree checkpoints are picked up, the others could be skipped.  In the POC, if the associated ingest table had more than 64k-100k of content, we didn't skip it.  This allows big changes to be picked up, so more space can be recovered via ingest pruning.

      We don't violate correctness by skipping a checkpoint pickup (even selectively) - all the needed data to retain correctly remains in the ingest tables, which are pruned according to what stable cursors remain open

      Two issues not fleshed out in the POC.  If we defer picking up a checkpoint for a Btree do we still pick up its metadata?  In the POC, we do not pick up the metadata.  I think as long as we are follower, that's okay, and even correct - we'll never persist that metadata.  However, I think the POC has a hole with step up.  Since the follower (using this optimization) never does a full pickup, it doesn't have the latest (full) checkpoint when stepping up.  So I think there's a final step when we're in this mode - when stepping up - whether or not we've seen the final checkpoint, we must reprocess it, and of course pick up every single btree's checkpoint that' changed from what we actually have.

      Part of the work here is also to measure before and after, and make sure all appropriate statistics are in place.

      A potential part of this work is to tune the N number (N=19 in the POC) and the content threshold (64k - 100k in the POC, purposely variable to avoid synchronizing pickups).  Tuning can also be done in a followup ticket.

            Assignee:
            Donald Anderson
            Reporter:
            Donald Anderson
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: