-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical - P2
-
None
-
Affects Version/s: None
-
Component/s: Schema Management
-
Storage Engines - Foundations
-
95.566
-
None
-
None
On a disaggregated leader, a table dropped at or below the stable schema epoch and recreated above it is resurrected after a crash and recovery, when it should be absent. Leader recovery reaches the same checkpoint pick-up reconciliation as a follower (__disagg_apply_checkpoint_meta() in src/conn/conn_layered_checkpoint_pick_up.c), which does not reconcile create/drop against last_disaggregated_schema_epoch.
Impact
- Phantom tables: A table whose latest durable schema operation at or below the stable epoch is a DROP is present after recovery. It is writable but out of step with the last durable schema state.
- Replica divergence: Other nodes recover to the stable epoch cleanly. The affected leader presents a schema inconsistent with the stable checkpoint.
- Btree ID conflict: The resurrected table can carry a stale btree ID (FIXME-WT-14730), matching the follower-side corruption in WT-18068.
Root cause
__disagg_apply_checkpoint_meta() is not epoch-aware. Its create decision restores any table present in the shared checkpoint without checking whether the latest schema operation at or below last_disaggregated_schema_epoch is actually a CREATE. The empty drop branch (FIXME-WT-17746) is the follower-side manifestation of the same gap.
A recreate above the stable epoch reaches the shared metadata by a side channel: writing a row to the recreated table forces its stable constituent to be checkpointed, so the table's file: entry lands in the shared checkpoint even though the CREATE itself is deferred (its epoch is above the stable schema epoch). Pick-up then restores it unconditionally.
Reproduction
Attached: test_layered_schema16.py. Deterministic, fails every run on current code.
- Create + publish table at epoch 2, checkpoint (durable).
- Drop + publish drop at epoch 3, checkpoint (removal durable). Both epochs at or below the stable epoch.
- Recreate + publish at epoch 9 (above stable), insert a row, so the stable constituent is checkpointed into shared metadata.
- Advance the stable schema epoch to 5 (between the drop at 3 and the recreate at 9), checkpoint. Durable epoch is now 5.
- Crash and recover with lose_all_my_data=true (the real leader recovery path, as in test/csuite/schema_disagg_abort).
- The last schema operation at or below the durable epoch (5) is the DROP at 3, so the table must be absent. Instead it is resurrected in local metadata.
Expected fix
Make the pick-up create/drop reconciliation epoch-aware: restore a table only when its latest schema operation at or below last_disaggregated_schema_epoch is a CREATE, and roll back creates whose epoch is above it. This is the leader/epoch variant of the follower gap in WT-18068 (empty drop branch, FIXME-WT-17746).
Related
WT-18068 (follower create/drop/create btree-ID corruption, same function and FIXME-WT-17746), WT-18101 (drop trims stable pages before the drop is durable), WT-18077 (parent).
- depends on
-
WT-18091 Handle schema operations before setting the stable schema epoch
-
- Open
-
- is related to
-
WT-18112 (Step-up) Follower-published table orphaned when stepping up before the first checkpoint
-
- Open
-
-
WT-18068 Follower reads stale btree ID after leader create/drop/create of a layered table
-
- In Code Review
-
-
WT-18077 Incorrect use of WT_ERR may bypass resource cleanup in bt_vrfy.c
-
- Closed
-
-
WT-17746 Handle table drop during checkpoint pick-up
-
- Open
-
-
WT-14730 Add some assertions when picking up new checkpoints
-
- Open
-
- related to
-
WT-18112 (Step-up) Follower-published table orphaned when stepping up before the first checkpoint
-
- Open
-
-
WT-18101 Replaying table drop() on an already discarded table's data fails
-
- Open
-
-
WT-18068 Follower reads stale btree ID after leader create/drop/create of a layered table
-
- In Code Review
-
-
WT-18093 Ensure that we clear WT_BTREE_AWAITS_PUBLISH correctly for recreated tables
-
- Needs Scheduling
-
-
WT-18092 Check if a new btree can be marked as published during eviction
-
- Needs Scheduling
-