Parallel checkpoint leaves worker transactions running when a checkpoint fails, tripping teardown assertion

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Checkpoints
    • None
    • Storage Engines - Persistence
    • 225.506
    • SE Persistence backlog
    • None

      Summary

      With parallel checkpoints enabled (checkpoint_threads>0), a checkpoint that fails part-way through leaves the reconciliation worker threads' transactions running. Nothing rolls them back on the failure path, so when the worker thread group is later torn down (at connection close), __checkpoint_parallel_thread_stop trips WT_ASSERT(session, !F_ISSET(session->txn, WT_TXN_RUNNING)) (src/checkpoint/checkpoint_parallel.c:265) — an abort in diagnostic builds, a WT_PANIC otherwise (src/checkpoint/checkpoint_parallel.c:267).

      Found via WT-18048's new test test_corrupt02 running under the parallel_checkpoint hook.

      Root cause

      Parallel-checkpoint reconciliation workers are per-connection (created in _wt_checkpoint_parallel_thread_create, destroyed at connection close in conn_open.c). Each worker begins a transaction for reconciliation and keeps it running; per the comment in checkpoint_parallel_thread_run, _"the main checkpoint thread is responsible for committing or rolling back this transaction." The coordinator commits them via _wti_checkpoint_parallel_commit, but only on the success path (src/checkpoint/checkpoint_txn.c:1934). There is no rollback counterpart. If the checkpoint errors before that commit (for example a corrupt extent-list read in ckpt_process returning WT_PANIC), the worker transactions remain WT_TXN_RUNNING, and the eventual wt_thread_group_destroy -> _checkpoint_parallel_thread_stop asserts or panics.

      Reproduce

      On a diagnostic build:

      python3 ../test/suite/run.py --hook parallel_checkpoint corrupt02
      

      The checkpoint itself fails cleanly (raises a handled WT_PANIC); the abort happens afterwards, on the checkpoint-page-reconciliation-threads during teardown. test_corrupt01 passes under the same hook because its corruption is hit on an ordinary read rather than inside a checkpoint, so it never fails a parallel checkpoint mid-flight.

      Note: this was uncovered because WT-18048 made the corrupt extent-list read handled gracefully instead of aborting on the (now removed) spurious __assert_ckpt_matches assertion; the parallel-checkpoint teardown gap it exposes is pre-existing.

      Proposed fix

      Add a rollback counterpart to _wti_checkpoint_parallel_commit — for example wti_checkpoint_parallel_rollback that walks the thread group and rolls back each worker's running transaction with wt_txn_rollback — and call it on the checkpoint error/cleanup path so no worker transaction is left running by the time the thread group is torn down. Alternatively, _checkpoint_parallel_thread_stop could roll back a running transaction instead of asserting, but an explicit rollback on the failure path keeps the invariant clear.

      Tasks

      • Add a worker-transaction rollback path for failed parallel checkpoints.
      • Invoke it from the checkpoint failure/cleanup path so worker transactions are not left running.
      • Re-enable test_corrupt02 under the parallel_checkpoint hook (remove the skip_for_hook stopgap) as the regression test.

      Definition of Done

      • A parallel checkpoint that fails part-way rolls back its worker transactions; connection close no longer asserts or panics in __checkpoint_parallel_thread_stop.
      • test_corrupt02 passes under --hook parallel_checkpoint without the skip.
      • No regressions in the checkpoint / salvage / corruption Python suites.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: