-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical - P2
-
None
-
Affects Version/s: None
-
Component/s: Schema Management
-
None
-
Storage Engines - Foundations
-
82.211
-
None
-
None
Summary
A layered table created and published on a follower is lost after the follower steps up to leader, when the step-up happens before any checkpoint has completed. The table's CREATE is durable by schema epoch, but it never reaches shared metadata, so it is absent after recovery.
Root cause
__layered_create_missing_stable_tables_helper() (src/conn/conn_layered.c) chooses between the epoch-aware path and the legacy no-epoch path from a single test:
stable_schema_epoch = conn->txn_global.last_ckpt_disaggregated_schema_epoch; if (stable_schema_epoch == WT_SCHEMA_EPOCH_NONE) return (__layered_create_missing_stable_tables_legacy(session));
last_ckpt_disaggregated_schema_epoch is WT_SCHEMA_EPOCH_NONE in two different situations: a database that does not use schema epochs, and a database that uses them but has not completed its first checkpoint. A follower never checkpoints, so at the first step-up the value is still NONE and the code wrongly takes the legacy path.
The legacy path clears the shared metadata queue, discarding the correctly epoch-ordered create/drop history published during the follower phase. It then rescans local metadata and re-enqueues a CREATE for each table missing its stable constituent, stamped with WT_SCHEMA_EPOCH_UNPUBLISHED (UINT64_MAX). Every checkpoint defers entries whose schema epoch is above the checkpoint's current epoch, and UINT64_MAX is above everything, so the entry is deferred on every checkpoint and never written to shared metadata. After recovery the table is missing.
Impact
- Data loss: a table published on a follower before the first checkpoint is silently lost after step-up and recovery, even though its CREATE is durable by the stable schema epoch.
- Affects the follower-then-leader startup sequence, such as a fresh cluster whose first node comes up as a follower and is then promoted.
Reproduction
Deterministic: test_disagg_stepup_create_loss.py. A leader never checkpoints; a follower created directly (no checkpoint to pick up) creates and publishes a layered table at schema epoch 20; the follower steps up to leader before any checkpoint completes; a checkpoint runs with the stable schema epoch advanced to 100. The table is absent from shared metadata, so a recovered node never sees it. Also reproduces intermittently in test/csuite/schema_disagg_abort switch mode.
Expected fix
Distinguish "no schema epochs" from "no checkpoint yet". The legacy path should be taken only for a genuine no-epoch database, not whenever the last checkpoint schema epoch is unset. When schema epochs are in use, step-up must replay the existing queue with the tables' real published epochs instead of clearing it and re-enqueuing at WT_SCHEMA_EPOCH_UNPUBLISHED.
- depends on
-
WT-18091 Handle schema operations before setting the stable schema epoch
-
- Open
-
- is related to
-
WT-18099 (Leader) create/drop/create above stable schema epoch survives database recovery
-
- Open
-
- related to
-
WT-18099 (Leader) create/drop/create above stable schema epoch survives database recovery
-
- Open
-
-
WT-18056 Disaggregated step-up re-enqueues follower-published CREATEs as unpublished
-
- Closed
-
-
WT-18093 Ensure that we clear WT_BTREE_AWAITS_PUBLISH correctly for recreated tables
-
- Needs Scheduling
-
-
WT-18092 Check if a new btree can be marked as published during eviction
-
- Needs Scheduling
-