Replaying table drop() on an already discarded table's data fails

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Schema Management
    • None
    • Storage Engines - Foundations
    • 94.522
    • None
    • None

      Not a trim-timing bug

      Trimming a layered table's stable pages before the drop becomes durable is by design. It prevents leaking the table's data if the leader crashes after the drop is durable but before the trim runs. The layered table contract makes this safe: after restarting from the newest checkpoint, the node replays exactly the operations the crashed leader performed but did not checkpoint, and two-phase drop guarantees a checkpoint between a table's final non-drop operation and its drop. So replay of a dropped table can only ever call drop() again, never a write. Deferring or epoch-gating the trim would violate that contract.

      Root cause

      drop() does not handle a layered table whose stable data has already been discarded by the page server.

      On restart from the newest checkpoint, the dropped table is restored from the shared checkpoint and drop() is replayed. The drop path opens the stable constituent's data handle to issue the trim (_drop_issue_trim -> _wt_session_get_dhandle in src/schema/schema_drop.c:139,162), which reads the constituent's root page. If the page server already reclaimed those pages, the read finds nothing and WiredTiger aborts:

      file:<name>.wt_stable, WT_SESSION.drop: __block_disagg_read_multiple(...), 166:
      WiredTiger assertion failed: '*results_count > 0'
      

      Why it usually does not bite

      The SLS page server delays trimming by ~24 hours, so during replay the stable pages almost always still exist and the drop succeeds. The correctness of drop should not rely on that delay.

      Reproduction

      Attached: test_disagg_drop_trim_data_loss.py. Palite's trim_table is a no-op, so the test stands in for a page server that has already reclaimed the pages by deleting the stable constituent's rows from palite's pages table while the connection is closed. It then restarts from the checkpoint and replays drop(), which aborts. Run in a subprocess so the abort does not take down the test runner.

      Fix

      Make the drop / trim path tolerate already-discarded stable data: issuing the trim must not require reading the constituent's pages, and a missing page image must be treated as already-trimmed rather than a fatal read failure.

      Related

      WT-18099 and WT-18068 (epoch-aware checkpoint pick-up), WT-18077 (parent).

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Jie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: