-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Critical - P2
-
None
-
Affects Version/s: None
-
Component/s: Schema Management
-
Storage Engines - Foundations
-
578.983
-
None
-
None
On a disaggregated leader, a table dropped at or below the stable schema epoch and recreated above it is resurrected after a crash and recovery, when it should be absent. Leader recovery reaches the same checkpoint pick-up reconciliation as a follower (__disagg_apply_checkpoint_meta() in src/conn/conn_layered_checkpoint_pick_up.c), which does not reconcile create/drop against last_disaggregated_schema_epoch.
Impact
- Phantom tables: A table whose latest durable schema operation at or below the stable epoch is a DROP is present after recovery. It is writable but out of step with the last durable schema state.
- Replica divergence: Other nodes recover to the stable epoch cleanly. The affected leader presents a schema inconsistent with the stable checkpoint.
- Btree ID conflict: The resurrected table can carry a stale btree ID (FIXME-WT-14730), matching the follower-side corruption in WT-18068.
Root cause
__disagg_apply_checkpoint_meta() is not epoch-aware. Its create decision restores any table present in the shared checkpoint without checking whether the latest schema operation at or below last_disaggregated_schema_epoch is actually a CREATE. The empty drop branch (FIXME-WT-17746) is the follower-side manifestation of the same gap.
A recreate above the stable epoch reaches the shared metadata by a side channel: writing a row to the recreated table forces its stable constituent to be checkpointed, so the table's file: entry lands in the shared checkpoint even though the CREATE itself is deferred (its epoch is above the stable schema epoch). Pick-up then restores it unconditionally.
Reproduction
Attached: test_layered_schema16.py. Deterministic, fails every run on current code.
- Create + publish table at epoch 2, checkpoint (durable).
- Drop + publish drop at epoch 3, checkpoint (removal durable). Both epochs at or below the stable epoch.
- Recreate + publish at epoch 9 (above stable), insert a row, so the stable constituent is checkpointed into shared metadata.
- Advance the stable schema epoch to 5 (between the drop at 3 and the recreate at 9), checkpoint. Durable epoch is now 5.
- Crash and recover with lose_all_my_data=true (the real leader recovery path, as in test/csuite/schema_disagg_abort).
- The last schema operation at or below the durable epoch (5) is the DROP at 3, so the table must be absent. Instead it is resurrected in local metadata.
Expected fix
Make the pick-up create/drop reconciliation epoch-aware: restore a table only when its latest schema operation at or below last_disaggregated_schema_epoch is a CREATE, and roll back creates whose epoch is above it. This is the leader/epoch variant of the follower gap in WT-18068 (empty drop branch, FIXME-WT-17746).
Related
WT-18068 (follower create/drop/create btree-ID corruption, same function and FIXME-WT-17746), WT-18101 (drop trims stable pages before the drop is durable), WT-18077 (parent).
- depends on
-
WT-18091 Handle schema operations before setting the stable schema epoch
-
- Open
-
-
WT-18093 Ensure that we clear WT_BTREE_AWAITS_PUBLISH correctly for recreated tables
-
- Closed
-
- is related to
-
WT-18068 Follower reads stale btree ID after leader create/drop/create of a layered table
-
- Blocked
-
-
WT-18112 Fix step-up losing follower-published table when no prior follower checkpoint
-
- Closed
-
-
WT-18077 Incorrect use of WT_ERR may bypass resource cleanup in bt_vrfy.c
-
- Closed
-
-
WT-17746 Handle table drop during checkpoint pick-up
-
- Open
-
-
WT-14730 Validate immutable metadata config fields on checkpoint pickup
-
- In Code Review
-
- related to
-
WT-18101 Replaying table drop() on an already discarded table's data fails
-
- Open
-
-
WT-18068 Follower reads stale btree ID after leader create/drop/create of a layered table
-
- Blocked
-
-
WT-18112 Fix step-up losing follower-published table when no prior follower checkpoint
-
- Closed
-
-
WT-18187 Optimize disaggregated publish check with a per-btree create_schema_epoch field
-
- Open
-
-
WT-18093 Ensure that we clear WT_BTREE_AWAITS_PUBLISH correctly for recreated tables
-
- Closed
-
-
WT-18176 Require the stable disaggregated schema epoch to be set before publishing
-
- Closed
-
-
WT-18092 Check if a new btree can be marked as published during eviction
-
- Open
-
-
WT-18230 Disagg checkpoint can expose a recreated table before it is visible
-
- In Code Review
-