Follower reads stale btree ID after leader create/drop/create of a layered table

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 609.541
    • SE Foundations - 2026-08-18
    • 5

      Summary

      On a disaggregated follower, a leader create -> drop -> create of the same layered table leaves the follower with a corrupt table handle. Reads of the recreated table fail with a bare WT_ERROR.

      Hits a pure follower: one that only picks up checkpoints and never publishes its own schema operations, so pickup relies entirely on diffing shared against local metadata. This is the realistic replica topology.

      Impact

      Ordinary drop-and-recreate of a namespace poisons every pure follower:

      • Best case: block manager catches the btree-ID mismatch and returns WT_ERROR; the recreated table is unreadable on the follower until it is rebuilt from scratch.
      • Worse case: on a read path with looser validation, the follower serves the old table's data under the new table's name with no error, a silent correctness violation.

      MongoDB drops and recreates namespaces routinely, so this affects data integrity on followers in the field.

      Root cause

      Two gaps in __disagg_apply_checkpoint_meta (src/conn/conn_layered_checkpoint_pick_up.c) combine on the same table name:

      1. Drop is a no-op on the follower (FIXME-WT-17746). The dropped table survives in local metadata with its original btree ID; the pickup drop branch is empty.
      2. Recreate reuses the stale entry (FIXME-WT-14730). Pickup copies the leader's new checkpoint address onto the stale entry but keeps the old id=, so the handle opens as the old btree while following the recreated btree's checkpoint.

      Reproduction

      Attached: test_layered_schema14.py. Asserts correct behavior, so it fails today and passes once the gaps are fixed.

      Fix

      Implementing the drop path (FIXME-WT-17746) is sufficient; the btree-ID conflict check (FIXME-WT-14730) is defense-in-depth.

        1. test_layered_schema14.py
          6 kB
        2. test_layered_schema15_no_epoch.py
          6 kB
        3. wt18068_single_pickup_repro.py
          5 kB

            Assignee:
            Yury Ershov
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: