(Leader) create/drop/create above stable schema epoch survives database recovery

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: Schema Management
    • Storage Engines - Foundations
    • 95.569
    • None
    • None

      On a disaggregated leader, a table dropped at or below the stable schema epoch and recreated above it is resurrected after a crash and recovery, when it should be absent. Leader recovery reaches the same checkpoint pick-up reconciliation as a follower (__disagg_apply_checkpoint_meta() in src/conn/conn_layered_checkpoint_pick_up.c), which does not reconcile create/drop against last_disaggregated_schema_epoch.

      Impact

      • Phantom tables: A table whose latest durable schema operation at or below the stable epoch is a DROP is present after recovery. It is writable but out of step with the last durable schema state.
      • Replica divergence: Other nodes recover to the stable epoch cleanly. The affected leader presents a schema inconsistent with the stable checkpoint.
      • Btree ID conflict: The resurrected table can carry a stale btree ID (FIXME-WT-14730), matching the follower-side corruption in WT-18068.

      Root cause

      __disagg_apply_checkpoint_meta() is not epoch-aware. Its create decision restores any table present in the shared checkpoint without checking whether the latest schema operation at or below last_disaggregated_schema_epoch is actually a CREATE. The empty drop branch (FIXME-WT-17746) is the follower-side manifestation of the same gap.

      A recreate above the stable epoch reaches the shared metadata by a side channel: writing a row to the recreated table forces its stable constituent to be checkpointed, so the table's file: entry lands in the shared checkpoint even though the CREATE itself is deferred (its epoch is above the stable schema epoch). Pick-up then restores it unconditionally.

      Reproduction

      Attached: test_layered_schema16.py. Deterministic, fails every run on current code.

      1. Create + publish table at epoch 2, checkpoint (durable).
      2. Drop + publish drop at epoch 3, checkpoint (removal durable). Both epochs at or below the stable epoch.
      3. Recreate + publish at epoch 9 (above stable), insert a row, so the stable constituent is checkpointed into shared metadata.
      4. Advance the stable schema epoch to 5 (between the drop at 3 and the recreate at 9), checkpoint. Durable epoch is now 5.
      5. Crash and recover with lose_all_my_data=true (the real leader recovery path, as in test/csuite/schema_disagg_abort).
      6. The last schema operation at or below the durable epoch (5) is the DROP at 3, so the table must be absent. Instead it is resurrected in local metadata.

      Expected fix

      Make the pick-up create/drop reconciliation epoch-aware: restore a table only when its latest schema operation at or below last_disaggregated_schema_epoch is a CREATE, and roll back creates whose epoch is above it. This is the leader/epoch variant of the follower gap in WT-18068 (empty drop branch, FIXME-WT-17746).

      Related

      WT-18068 (follower create/drop/create btree-ID corruption, same function and FIXME-WT-17746), WT-18101 (drop trims stable pages before the drop is durable), WT-18077 (parent).

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: