A schema operation can be published below the node's last checkpoint schema epoch

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 153.624
    • None
    • 1

      A checkpoint stamped with schema epoch E claims to cover every schema operation published at or below E. Both paths that remove entries from the shared metadata operations queue rely on that: a leader's _wt_disagg_shared_metadata_queue_process drains entries at or below the epoch of the checkpoint it is taking, and a follower's _wti_disagg_shared_metadata_queue_prune removes entries at or below the epoch of the checkpoint it has just adopted.

      WT_SESSION::publish does not enforce that contract. __schema_publish_disagg_schema_epoch (src/schema/schema_publish.c) rejects an epoch at or below the stable schema epoch and at or below the step-down boundary, but it never compares against the node's last written or adopted checkpoint epoch (last_ckpt_disaggregated_schema_epoch). An operation published below that watermark is behind it from the moment it exists: no checkpoint will claim to introduce it, yet the checkpoint that has already been taken claims to have covered it.

      Such an entry is then removed only if the node happens to take a checkpoint as leader or complete another pickup as a follower. If it steps up before either, the queue walk in __layered_create_missing_stable_tables_helper finds it and aborts on its own invariant:

      WiredTiger assertion failed: 'entry->schema_epoch > stable_schema_epoch ||
      stable_schema_epoch == 0'
      

      whose comment reads "when the stable epoch is known, entries older than it should have been pruned".

      Evidence

      Instrumenting the epoch assignment in __wt_disagg_shared_metadata_queue_publish alongside last_ckpt_disaggregated_schema_epoch, and the queue walk at step-up, shows the publish stamping an epoch well below the watermark and the same entry tripping the assert later:

      publish-stamp    table=schema_0_3_56 op=CREATE epoch=744 last_ckpt=813
      stale-at-step-up table=schema_0_3_56 op=CREATE epoch=744 stable=813
      

      Reproduced in 2 of 3 parallel workers of:

      ./test_schema_disagg_abort -b <build> -r lf -s 3 -t 25 -T 8 -u 64
      

      This needs the WT-18068 work in place: without it the same run dies earlier on the create/drop/create btree ID panic.

      Two other explanations were investigated and ruled out, so they need not be re-checked:

      • __disagg_requeue_skipped_creates returning parked CREATE entries with their original epochs - there were no requeues at all in the failing runs.
      • the one-checkpoint entry->deferred delay in queue_process - the entries that tripped the assert were never deferred.

      Note that stepping up does not adopt the latest checkpoint: __disagg_step_up discards a pending deferred pickup ("a leader never adopts checkpoints"), so the window between publishing below the watermark and stepping up is easy to hit.

      Possible fixes

      1. Reject the publish. Extend the guard in __schema_publish_disagg_schema_epoch to fail when the requested epoch is at or below last_ckpt_disaggregated_schema_epoch, in the same shape as the existing stable-epoch and step-down-boundary checks. The caller in test/csuite/schema_disagg_abort publishes at epochs derived from its own frontier, which lags after a role switch, so it would also need to publish above the watermark it can read through query_timestamp(get=last_disaggregated_schema_epoch).
      2. If replication can legitimately assign an epoch below a node's last adopted checkpoint epoch, then the first option is wrong and the queue removal paths cannot key on the epoch at all: they would have to decide from whether the operation actually reached shared metadata.

      Deciding between those needs an answer on what epochs replication assigns around a step-up, which is why this is filed rather than patched.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: