schema_disagg_abort aborts when checkpoint pick-up already applied a relayed drop

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 119.491
    • None
    • None

      Symptom

      test_schema_disagg_abort aborts on the follower node with a missing table on a relayed drop:

      FAILED: schema_op_execute: DROP table:schema_0_9_6: No such file or directory
      node1 terminated unexpectedly: signal 6
      

      Always the follower, always a table relayed from the leader. Reproduced 6 of 8 runs of -r lf -T 12 -u 64 -t 120 on a build carrying the layered checkpoint pick-up local-metadata removal.

      Mechanism

      A single leader drop reaches the follower by two independent routes:

      1. Checkpoint pick-up sees the table missing from shared metadata, takes the md_has[LAYERED] && !sh_has[LAYERED] branch, and removes the table's local metadata entries, including table:T.
      2. The relayed schema event then applies its own drop of that same table, which is now gone, and gets ENOENT.

      Instrumenting both discard sites gave a direct hit on adjacent log lines, same table, with the checkpoint epoch equal to the drop's timestamp:

      discard "schema_0_9_6" latest_op=NONE latest_epoch=0 ckpt_epoch=1693 startup=0
      FAILED: DROP table:schema_0_9_6 (ts 1693): No such file or directory
      

      The two paths both run under the schema lock, and the test drops with lock_wait=false and retries, so they do not interleave. This is not a race. It is double application of one logical operation: the second path does redundant work and returns an error instead of being a no-op.

      Fix

      The test is the application here, and the follower's shared checkpoint is authoritative for table existence, so a drop of an already-absent table is a success state for the workload. schema_op_execute treats ENOENT on a drop as success.

      Note on reachability

      The abort is only reachable once the pick-up-side local-metadata removal lands (WT-18068). On current develop the pick-up leaves the local entries behind, so the change is defensive there and cannot be validated by a repro on develop alone.

            Assignee:
            Jie Chen
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: