ExportXMLWordPrintableJSON

    • Storage Engines - Foundations
    • 27.893
    • None
    • None
    • 0

      Symptom

      schema-disagg-abort-legacy-switch-test-disagg-1 on amazon2023-disagg-asan-stress (wiredtiger-disagg), 2 of 60 runs between 2026-09-10 and 2026-09-24. It has not failed on the plain variant (60/60).

      test_schema_disagg_abort: FAILED: node0 leader DROP table:schema_0_1_5_0: EBUSY for 30 seconds: the table has dirty data and cannot be closed yet: Connection timed out
      Error at src/checkpoint/checkpoint_txn.c:3588: "ret" failed with Device or resource busy (16), WT_DIRTY_DATA: Table has dirty data
      
      Date Revision Iteration CONFIG
      2026-09-11 1e0bcde8b0 9/10 -e -r l -s 30 -u 64 -T 12 -t 300 -PSD827050,E12579209
      2026-09-17 3069b317e5 8/10 -e -r l -s 30 -u 64 -T 12 -t 300 -PSD13916421,E8793070

      Both revisions already contain both earlier fixes for this message:

      • c51c88b966 (WT-18415): legacy schema ops now take their timestamp after they succeed, so a blocked drop can no longer freeze the frontier.
      • 9fb06a4008 (WT-18515): closed WT-18529.

      The WT-18529 auto-resolution rule did not match either run, so BFG-3644409 and BFG-3658813 are open and unlinked.

      Not the WT-18529 failure: checkpoints stop

      Both failing iterations print one periodic checkpoint before the first role switch and one after it, then none for the 30 s the drop waits:

      Node 0: checkpoint 1 at 51 (lsn 15)
      Parent: directing switch 1
      Node 0: checkpoint 2 at 534 (lsn 209)
      Node 0: stable 735, frontier 735, last checkpoint 579, step-down 0
      

      A passing run of the same task (2026-09-24, 6f6e84ab46) prints 8-10 checkpoints before switch 1 and 5 between switches 1 and 2. WT-18529's 2026-08-28 log printed 8 before switch 1. The cadence is 1-4 s (ckpt.c:246), and with step-down 0 the leader is still on the periodic path (ckpt.c:293-296).

      The stable timestamp keeps up with the frontier, but the last checkpoint timestamp stops advancing. So the checkpoint is stalled, not the stable timestamp. The drop's EBUSY is a symptom.

      Open question

      What stalls the checkpoint? One candidate: up to 12 workers retrying drop(force=false,lock_wait=true) every 10 ms (worker.c:133-168) starve it of the locks it needs. Nothing verifies that yet. The node core from the 09-17 run would show the checkpoint thread's stack. It is in the task artifact: WT_TEST.test_schema_disagg_abort/dump_test_schema_dis.16937.core.

      CI

      WT-18737 moves schema_disagg_abort to the main project and excludes legacy-switch on amazon2023-arm64-asan under a FIXME pointing at this ticket.

            Assignee:
            Alex Blekhman
            Reporter:
            Alex Blekhman
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: