During disagg step-up on the new primary, a drain worker failed to clear an ingest table and the WT_THREAD_PANIC_FAIL thread group turned it into a panic:
2026-08-27T19:17:22.636 [WT] __layered_drain_worker_run:856: Failed to clear ingest table
"file:collection-96883c24-fca0-4443-93ed-f1eb93f2fee7.wt_ingest":
WT_ROLLBACK: conflict between concurrent operations
[WT] __thread_run:47: Unrecoverable utility thread error
[WT] __wt_panic ... -> mongod fassert 50853 (wiredtiger_util.cpp) -> node aborted
Initial investigation shows a bug of self-inflicted sequence when truncate the ingest table, no concurrent writer required:
- Tombstone key K (globally visible), call next().
- next() returns WT_RESTART — e.g. an in-memory split (the tombstones themselves grow page footprint), or the page being evicted/pruned under the cursor (the ingest btree is WT_BTREE_GARBAGE_COLLECT; the clear's own globally-visible tombstones make pages prunable, and the drain typically runs with the cache heavily dirty).
- Retry re-searches K, which is now invisible (its own tombstone) or physically pruned, so search returns WT_NOTFOUND.
- The final mapping converts that to WT_ROLLBACK; the drain worker error panics the process.
The NOTFOUND-to-ROLLBACK mapping is correct for transactional truncates (start key concurrently removed = retryable conflict), but for the non-transactional clear the truncate's own globally-visible writes can trigger it.
This ticket is to verify this bug, propose a solution and fix it.
- related to
-
WT-18614 Disagg drain worker panics when the ingest table's stable pair is gone from the metadata
-
- Needs Scheduling
-