Lazily open the stable table on follower to fix lost tombstone on blind-delete miss (dup-key fasserts)

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • WT12.0.0, 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: DHandles
    • None
    • Storage Engines, Storage Engines - Foundations
    • 733.183
    • Storage Execution 2026-08-03
    • None

      Per investigations in WT-17617, a large part of the DSC-vs-ASC regression came from repeatedly re-opening stable-table dhandles on the follower every time the system picks up a new checkpoint. Given blind writes, we may not always need the stable handle at all, so this ticket lazily opens the stable table only when actually required, rather than eagerly on every checkpoint pickup.

      Correctness bug this also fixes: WT-17254's blind-delete path decides up front, before the operation runs, whether to open the stable table dhandle – and skips it when it expects to take the blind-delete branch, trusting the caller's overwrite=true guarantee that the key is live somewhere. But the operation can still end up taking the normal (non-blind) delete path instead, and that path does need stable to resolve the key. Because stable was never opened – the earlier decision assumed blind-delete and skipped it – the normal delete has no way to find a key that exists only in stable (not in ingest): it surfaces as "not found" and no tombstone is written anywhere. Mongo tolerates the "not found" result and continues. A later reinsertion of the same key then collides with the never-tombstoned original in stable, producing WT_DUPLICATE_KEY / NoSuchKey fasserts on followers/standbys during oplog application.

      This ticket's change defers the decision of whether to open the stable table to the point of the actual delete operation, instead of deciding eagerly beforehand. As a result, the normal (non-blind) delete correctly opens stable when needed and writes the tombstone, closing the gap left by WT-17254. The fix is a side effect of the lazy-open redesign, not a targeted bug fix – but it does address the root cause (missing stable lookup on blind-delete miss), not just a symptom.

      Observed impact: HELP-97958, SERVER-132631, BF-44855 – clusters running RC1017 (which has WT-17254 but not this fix) hit duplicate-key/fassert errors after index key removal + reinsertion.

      Regression coverage: added later in WT-18215 (PR #14325 – https://github.com/wiredtiger/wiredtiger/pull/14325) – this test fails when run against the pre-WT-18159 (RC1017) binary, confirming the fix is what closes the gap.

            Assignee:
            Wei Hu
            Reporter:
            Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: