-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
None
-
Storage Engines - Foundations
-
79.791
-
None
-
None
Background
WT-17254 added a fast path for blind removes on a follower's ingest table: with overwrite=true on WT_CURSOR::remove, the stable-table lookup is skipped and only the ingest table + truncate list are consulted. This shipped in the rc1017 disagg release and gave a measurable standby-performance win (e.g. tpcc_majority_out_of_cache / tpcc_majority_agg_out_of_cache on BF-43334).
After upgrading clusters to rc1019, a new set of AFs appeared, symptomatic of blind removes being applied to a follower's ingest table targeting keys that have already been removed:
- AF-19942 - __clayered_remove_from_ingest:2792: WiredTiger assertion failed: '0'. overwrite=true should guarantee the key exists for remove
- AF-19923 - __layered_assert_stable_btree_state:77: WiredTiger assertion failed: 'has_value || __wt_txn_upd_visible_all(session, last_upd)'. No corresponding value exists on the stable table to delete
There is no confirmed root cause for these AFs yet. As a temporary mitigation, WT-18390 reverted / disabled the blind remove path in __clayered_remove_from_ingest so this class of issue does not show up in the next disagg release.
Scope
Re-enable the blind remove path disabled by WT-18390 (i.e. restore the WT-17254 behavior), only once the root cause of AF-19942 and AF-19923 has been determined and a fix is in place that is compatible with blind removes.
Definition of done
- Root cause of AF-19942 and AF-19923 confirmed and fixed.
- Blind remove path in __clayered_remove_from_ingest re-enabled (
WT-18390reverted). - Validated against the workloads/AFs that originally exposed the corruption.