-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
Storage Engines - Foundations
-
265.602
-
None
-
None
A checkpoint stamped with schema epoch E claims to cover every schema operation published at or below E, and both paths that remove entries from the shared metadata operation queue key on that: a leader's _wt_disagg_shared_metadata_queue_process drains entries at or below the epoch of the checkpoint it is taking, and a follower's _wti_disagg_shared_metadata_queue_prune removes entries at or below the epoch of the checkpoint it has just adopted.
WT_SESSION::publish does not enforce that contract. _schema_publish_disagg_schema_epoch rejects an epoch at or below the stable schema epoch, and _wt_disagg_shared_metadata_queue_publish rejects one at or below the step-down boundary while that boundary is set, but nothing compares against last_ckpt_disaggregated_schema_epoch.
Why only followers are exposed
__wt_txn_global_set_timestamp (txn_timestamp.c) already refuses to set a stable schema epoch below the epoch recorded in the last checkpoint, but that check is gated on the node being a leader:
if (has_stable_disagg_epoch &&
__wt_atomic_load_bool_relaxed(&S2C(session)->layered_table_manager.leader) &&
stable_disagg_epoch < last_ckpt_disagg_epoch) {
On a leader, publish requires an epoch above the stable epoch and the stable epoch is floored at the last checkpoint's, so a publish is always above the checkpoint epoch and the contract holds implicitly. A follower is excluded from that floor: checkpoint pickup advances the adopted epoch without advancing the stable epoch, which legitimately lags (checkpoint_txn.c notes "a follower's epoch legitimately sits below the last checkpoint's"). That leaves the range above the stable epoch and at or below the adopted checkpoint epoch, where a publish passes every existing check.
Consequence
An operation published in that range is behind every removal path from the moment it exists. It is either discarded by a later prune as "covered" without ever reaching shared metadata, so the operation is lost on that node, or the node steps up before any prune or leader checkpoint and the queue walk in __layered_create_missing_stable_tables_helper aborts on its own invariant that queue entries sit above the last adopted checkpoint epoch.
Instrumenting the epoch assignment in __wt_disagg_shared_metadata_queue_publish alongside last_ckpt_disaggregated_schema_epoch showed the publish stamping an epoch below the adopted one and the same entry tripping the assertion later:
publish-stamp table=schema_0_3_56 op=CREATE epoch=744 last_ckpt=813 stale-at-step-up table=schema_0_3_56 op=CREATE epoch=744 stable=813
Proposed fix
Reject the publish when the requested epoch is at or below last_ckpt_disaggregated_schema_epoch, in the same shape as the existing stable-epoch and step-down-boundary checks, scoped to entries being stamped now so that replays of already-published entries still pass. This completes on the follower the rule the engine already enforces on the leader rather than introducing a new one.
The queue prune in checkpoint pickup must also move below the adoption bookkeeping, so the adopted epoch is published before the prune runs. Otherwise a publish landing between the prune and the epoch store is neither rejected nor covered, and strands its entry.
Open question
This was originally raised as WT-18370 and deliberately not patched there, pending an answer to what epochs replication assigns around a step-up. If replication can legitimately assign an epoch below a node's last adopted checkpoint epoch, then rejecting the publish is the wrong fix and the queue removal paths cannot key on the epoch at all: they would have to decide from whether the operation actually reached shared metadata. That question gates this work.
Separately, mongod wraps WiredTiger calls in invariantWTOK, so a publish returning EINVAL crashes the node rather than surfacing an error. The argument that this is unreachable in production rests on the same assumption as above.