(Step-up) Follower-published table orphaned when stepping up before the first checkpoint

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: Schema Management
    • None
    • Storage Engines - Foundations
    • 82.237
    • None
    • None

      Summary

      A layered table created and published on a follower is lost after the follower steps up to leader, when the step-up happens before any checkpoint has completed. The table's CREATE is durable by schema epoch, but it never reaches shared metadata, so it is absent after recovery.

      Root cause

      __layered_create_missing_stable_tables_helper() (src/conn/conn_layered.c) chooses between the epoch-aware path and the legacy no-epoch path from a single test:

      stable_schema_epoch = conn->txn_global.last_ckpt_disaggregated_schema_epoch;
      if (stable_schema_epoch == WT_SCHEMA_EPOCH_NONE)
          return (__layered_create_missing_stable_tables_legacy(session));
      

      last_ckpt_disaggregated_schema_epoch is WT_SCHEMA_EPOCH_NONE in two different situations: a database that does not use schema epochs, and a database that uses them but has not completed its first checkpoint. A follower never checkpoints, so at the first step-up the value is still NONE and the code wrongly takes the legacy path.

      The legacy path clears the shared metadata queue, discarding the correctly epoch-ordered create/drop history published during the follower phase. It then rescans local metadata and re-enqueues a CREATE for each table missing its stable constituent, stamped with WT_SCHEMA_EPOCH_UNPUBLISHED (UINT64_MAX). Every checkpoint defers entries whose schema epoch is above the checkpoint's current epoch, and UINT64_MAX is above everything, so the entry is deferred on every checkpoint and never written to shared metadata. After recovery the table is missing.

      Impact

      • Data loss: a table published on a follower before the first checkpoint is silently lost after step-up and recovery, even though its CREATE is durable by the stable schema epoch.
      • Affects the follower-then-leader startup sequence, such as a fresh cluster whose first node comes up as a follower and is then promoted.

      Reproduction

      Deterministic: test_disagg_stepup_create_loss.py. A leader never checkpoints; a follower created directly (no checkpoint to pick up) creates and publishes a layered table at schema epoch 20; the follower steps up to leader before any checkpoint completes; a checkpoint runs with the stable schema epoch advanced to 100. The table is absent from shared metadata, so a recovered node never sees it. Also reproduces intermittently in test/csuite/schema_disagg_abort switch mode.

      Expected fix

      Distinguish "no schema epochs" from "no checkpoint yet". The legacy path should be taken only for a genuine no-epoch database, not whenever the last checkpoint schema epoch is unset. When schema epochs are in use, step-up must replay the existing queue with the tables' real published epochs instead of clearing it and re-enqueuing at WT_SCHEMA_EPOCH_UNPUBLISHED.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: