Record the awaiting-publication state in the metadata table

    • Storage Engines - Foundations
    • 22.134
    • None
    • None

      Problem

      A layered table created while a stable disaggregated schema epoch is set awaits publication, and its stable constituent carries WT_BTREE_AWAITS_PUBLISH to keep the tree out of every checkpoint until the table is published.

      The flag is derived once, at handle open, in _btree_conf (src/btree/bt_handle.c:629). It requires the WT_SESSION_CREATE_BTREE session marker, which is set only for the duration of _create_file (src/schema/schema_create.c:396-408), so every later open takes the else branch and clears the flag. Unlike the in-memory configuration, which is re-derived from the file's own in_memory configuration on every open, there is no durable record of "not yet published" for the btree layer to read.

      Any close and reopen inside the publication window therefore silently promotes an unpublished table to published: it is checkpointed, its pages are written out, and the guard in __checkpoint_disagg_maybe_publish (src/checkpoint/checkpoint_txn.c:453-459) cannot fire because the state it tests has been erased.

      WT-18311 closed the one reachable path by making the sweep server decline to close a tree that awaits publication. That is a targeted guard, not a fix for the underlying fragility: every other close and reopen path has the same hole, and each one has to be found and guarded individually.

      Suggested work

      Persist the awaiting-publication state so that __btree_conf can re-derive it, rather than depending on the handle staying resident. Two directions worth comparing:

      • A durable marker in the stable constituent's local metadata row, cleared when a checkpoint publishes the table. This survives restart as well as a handle close.
      • Re-derivation from the in-memory shared metadata queue, which already answers "is there a pending CREATE for this table above the stable schema epoch" (__wti_disagg_table_latest_create_remove, src/conn/conn_layered.c:461). Cheaper, but it does not survive restart and needs care so that a follower opening a relayed table does not acquire the flag.

      Note that btree->min_unpublished_durable_ts is per-handle state in the same family and is lost by the same closes.

      Once the state is recoverable, the sweep guard added by WT-18311 can be revisited.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: