-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Layered Tables
-
Storage Engines - Foundations
-
98.429
-
None
-
None
WT-17254 added a fast path for blind removes on the follower's ingest table. This path skips the lookup of the stable table if we are in overwrite mode, and only checks the ingest table + truncate list. This change was included in the rc1017 disagg release.
Since rc1017 was deployed, we have seen a data-corruption bug caused by a gap in this blind remove path, tracked variously in HELP-97958, SERVER-132631, BF-44855. The corresponding fix for this, WT-18159, was merged and included in the rc1019 release.
After upgrading clusters to rc1019, we immediately saw a new set of AFs (AF-19942 and AF-19923), exposed by WT-18159. They are symptomatic of blind removes being applied to a follower's ingest table, targeted at keys that have already been removed.
At the time of writing, we don't have a confirmed root cause - the immediate response was to disable blind writes on the affected clusters (which stopped the AFs), and downgrade back to the relatively stable rc1018 build.
This ticket is to selectively disable this blind remove path in WiredTiger, until we have determined a root cause and a fix for AF-19942 and AF-19923 that is compatible with blind removes.
Once this is complete, the next steps are to cherry-pick it into the latest dsc-release branch and include it in the next disagg release build.
- is related to
-
WT-17254 Require a caller-guaranteed live key for overwrite=true remove on a layered follower cursor
-
- Closed
-
-
WT-18404 Re-enable blind ingest remove path in once it is safe to do so
-
- Needs Scheduling
-
-
WT-18159 Lazily open the stable table on follower to fix lost tombstone on blind-delete miss (dup-key fasserts)
-
- Closed
-
- related to
-
WT-18404 Re-enable blind ingest remove path in once it is safe to do so
-
- Needs Scheduling
-