-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Checkpoints
-
None
-
Storage Engines - Persistence
-
225.506
-
SE Persistence backlog
-
None
Summary
With parallel checkpoints enabled (checkpoint_threads>0), a checkpoint that fails part-way through leaves the reconciliation worker threads' transactions running. Nothing rolls them back on the failure path, so when the worker thread group is later torn down (at connection close), __checkpoint_parallel_thread_stop trips WT_ASSERT(session, !F_ISSET(session->txn, WT_TXN_RUNNING)) (src/checkpoint/checkpoint_parallel.c:265) — an abort in diagnostic builds, a WT_PANIC otherwise (src/checkpoint/checkpoint_parallel.c:267).
Found via WT-18048's new test test_corrupt02 running under the parallel_checkpoint hook.
Root cause
Parallel-checkpoint reconciliation workers are per-connection (created in _wt_checkpoint_parallel_thread_create, destroyed at connection close in conn_open.c). Each worker begins a transaction for reconciliation and keeps it running; per the comment in checkpoint_parallel_thread_run, _"the main checkpoint thread is responsible for committing or rolling back this transaction." The coordinator commits them via _wti_checkpoint_parallel_commit, but only on the success path (src/checkpoint/checkpoint_txn.c:1934). There is no rollback counterpart. If the checkpoint errors before that commit (for example a corrupt extent-list read in ckpt_process returning WT_PANIC), the worker transactions remain WT_TXN_RUNNING, and the eventual wt_thread_group_destroy -> _checkpoint_parallel_thread_stop asserts or panics.
Reproduce
On a diagnostic build:
python3 ../test/suite/run.py --hook parallel_checkpoint corrupt02
The checkpoint itself fails cleanly (raises a handled WT_PANIC); the abort happens afterwards, on the checkpoint-page-reconciliation-threads during teardown. test_corrupt01 passes under the same hook because its corruption is hit on an ordinary read rather than inside a checkpoint, so it never fails a parallel checkpoint mid-flight.
Note: this was uncovered because WT-18048 made the corrupt extent-list read handled gracefully instead of aborting on the (now removed) spurious __assert_ckpt_matches assertion; the parallel-checkpoint teardown gap it exposes is pre-existing.
Proposed fix
Add a rollback counterpart to _wti_checkpoint_parallel_commit — for example wti_checkpoint_parallel_rollback that walks the thread group and rolls back each worker's running transaction with wt_txn_rollback — and call it on the checkpoint error/cleanup path so no worker transaction is left running by the time the thread group is torn down. Alternatively, _checkpoint_parallel_thread_stop could roll back a running transaction instead of asserting, but an explicit rollback on the failure path keeps the invariant clear.
Tasks
- Add a worker-transaction rollback path for failed parallel checkpoints.
- Invoke it from the checkpoint failure/cleanup path so worker transactions are not left running.
- Re-enable test_corrupt02 under the parallel_checkpoint hook (remove the skip_for_hook stopgap) as the regression test.
Definition of Done
- A parallel checkpoint that fails part-way rolls back its worker transactions; connection close no longer asserts or panics in __checkpoint_parallel_thread_stop.
- test_corrupt02 passes under --hook parallel_checkpoint without the skip.
- No regressions in the checkpoint / salvage / corruption Python suites.
- is related to
-
WT-18048 Use-after-free of cached checkpoint list in __wti_block_checkpoint_extlist_dump
-
- Closed
-