Step-down clears the shared metadata queue instead of pruning it, losing publishes above the step-down timestamp

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Schema Management
    • None
    • Storage Engines
    • 22.219
    • None
    • None

      __disagg_step_down_int clears the whole shared metadata queue (conn_layered.c:1207). By that point the step-down checkpoint has already consumed every entry at or below the step-down timestamp T, so the clear's only effect is destroying entries above T — publishes made during the step-down window, and pending WT_SCHEMA_EPOCH_UNPUBLISHED REMOVEs.

      With async step-down (WT-17895) writes are expected to continue above T. Their data goes to ingest and survives the demotion until a covering checkpoint is adopted; the queue entries describing the same operations are silently destroyed. The publish itself returns success — nothing rejects it while the step-down timestamp is set.

      The pick-up path already implements the intended lifecycle: __wti_disagg_shared_metadata_queue_prune removes only entries at or below the adopted epoch (conn_layered_checkpoint_pick_up.c:1197). Follower publish (WT-17089) depends on post-boundary entries surviving a role transition — that is how a mirrored publish becomes durable after promotion. Step-down is the one place that breaks the symmetry.

      This looks like a leftover: the clear predates async step-down (WT-16565, Feb 2026), from when a step-down only ran fully quiesced and clear was equivalent to prune. Its comment ("abandoning the current checkpoint") describes a state that no longer exists at that point.

      Impact: masked while the peer's mirrored queue entry drains after promotion; real when the demoting node is the eventual survivor — it loses an acknowledged publish with no way to recover it, and can later hit "stable data checkpointed for unpublished table" (checkpoint_txn.c:454-459) once stable passes the orphaned ingest data.

      Suggested fix: replace the clear with __wti_disagg_shared_metadata_queue_prune(session, <step-down checkpoint schema epoch>). The other clear call sites (connection destroy, legacy no-epoch rebuild) are correct as-is.

      Found during WT-18258 (running operations during the step-down window in schema_disagg_abort).

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Alex Blekhman
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: