-
Type:
Improvement
-
Resolution: Won't Do
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Checkpoints
-
None
-
Storage Engines, Storage Engines - Foundations
-
359.053
-
None
-
None
This comes out of the investigation in WT-17617, where this idea was POC'd. In the data_handle workload, there are a lot of stable btrees updated in each checkpoint. Rather than picking up all of them at once, we can choose which ones we want. For example, for a given checkpoint, suppose that 1/Nth of the btree checkpoints are picked up, the others could be skipped. In the POC, if the associated ingest table had more than 64k-100k of content, we didn't skip it. This allows big changes to be picked up, so more space can be recovered via ingest pruning.
We don't violate correctness by skipping a checkpoint pickup (even selectively) - all the needed data to retain correctly remains in the ingest tables, which are pruned according to what stable cursors remain open
Two issues not fleshed out in the POC. If we defer picking up a checkpoint for a Btree do we still pick up its metadata? In the POC, we do not pick up the metadata. I think as long as we are follower, that's okay, and even correct - we'll never persist that metadata. However, I think the POC has a hole with step up. Since the follower (using this optimization) never does a full pickup, it doesn't have the latest (full) checkpoint when stepping up. So I think there's a final step when we're in this mode - when stepping up - whether or not we've seen the final checkpoint, we must reprocess it, and of course pick up every single btree's checkpoint that' changed from what we actually have.
Part of the work here is also to measure before and after, and make sure all appropriate statistics are in place.
A potential part of this work is to tune the N number (N=19 in the POC) and the content threshold (64k - 100k in the POC, purposely variable to avoid synchronizing pickups). Tuning can also be done in a followup ticket.
- is related to
-
WT-17617 Investigate >60% OperationThroughput regression in DSC vs ASC for data_handle_locust workload
-
- Closed
-
-
WT-18170 Dhandle scaling Perf: cooperative on demand background open of stable tables
-
- Backlog
-
-
WT-18159 Lazily open the stable table on follower
-
- In Code Review
-
- related to
-
WT-18061 Dhandle scaling Perf: On standby, layered cursors can pick up new stable checkpoints with staggered delays
-
- Open
-