-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
None
-
Storage Engines - Foundations
-
None
-
None
On layered tables, which don't support fast truncate, provide a temporary "best effort" workaround to avoid having large truncates fail if they touch more data than fits in the cache.
- Allow best-effort=true config option for WT_SESSION::truncate() call.
- When this option is set and WT uses slow truncate, WT starts truncating from the start cursor, and stops either when it reaches the end cursor or when it has truncated a fixed number of records (or amount of data)
- WT returns success if it ends early because it hit the maximum number of records (or data).
- WT does not need to tell the caller how far it got in a "best effort" truncate.
MOTIVATION
Without fast truncate support, the server sometimes sees truncate failures on the oplog, apparently because the amount of truncated data exceeds what WT can handle in a single transaction. This server will make some changes to reduce how much it truncates in a single call, but requested this change as well to handle any large truncates that still slip through to WT.
COMMENTS
Adding the best-effort config option (or something similar) lets us make this change without breaking all of the existing WT testing. We'll still need an ad hoc test to verify that best-effort works.
It would be preferable to set the truncation limit based on the total size of truncated records with a limit of, say 10% of the dirty eviction target. But if that isn't simple enough, we could assume a max record size of, say 1MB, and then use the same math to figure out how many records to truncate.
I think the required work here is
- Plumbing the config option to the truncate command
- Adding the best-effort limit to __wt_range_truncate()
- Adding a python test
This behavior is only needed until we have fast truncate support for layered tables on both leaders and followers. At that point this change should be reverted.