Re-enable blind ingest remove path in once it is safe to do so

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Layered Tables
    • None
    • Storage Engines - Foundations
    • 79.791
    • None
    • None

      Background

      WT-17254 added a fast path for blind removes on a follower's ingest table: with overwrite=true on WT_CURSOR::remove, the stable-table lookup is skipped and only the ingest table + truncate list are consulted. This shipped in the rc1017 disagg release and gave a measurable standby-performance win (e.g. tpcc_majority_out_of_cache / tpcc_majority_agg_out_of_cache on BF-43334).

      After upgrading clusters to rc1019, a new set of AFs appeared, symptomatic of blind removes being applied to a follower's ingest table targeting keys that have already been removed:

      • AF-19942 - __clayered_remove_from_ingest:2792: WiredTiger assertion failed: '0'. overwrite=true should guarantee the key exists for remove
      • AF-19923 - __layered_assert_stable_btree_state:77: WiredTiger assertion failed: 'has_value || __wt_txn_upd_visible_all(session, last_upd)'. No corresponding value exists on the stable table to delete

      There is no confirmed root cause for these AFs yet. As a temporary mitigation, WT-18390 reverted / disabled the blind remove path in __clayered_remove_from_ingest so this class of issue does not show up in the next disagg release.

      Scope

      Re-enable the blind remove path disabled by WT-18390 (i.e. restore the WT-17254 behavior), only once the root cause of AF-19942 and AF-19923 has been determined and a fix is in place that is compatible with blind removes.

      Definition of done

      • Root cause of AF-19942 and AF-19923 confirmed and fixed.
      • Blind remove path in __clayered_remove_from_ingest re-enabled (WT-18390 reverted).
      • Validated against the workloads/AFs that originally exposed the corruption.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Alexander Pullen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: