Context

This bug was discovered during testing on the dedicated elegant step-down feature branch (https://github.com/wiredtiger/wiredtiger/compare/develop...wt-17785-enable-elegant-stepdown-mainine). Currently, step-down restarts for test/format. In this branch, we are replacing the restart with a synchronous, elegant step-down triggered via reconfigure(role=follower). This ticket captures one of the bugs as a result of elegant stepdown.

Root Cause

During step-up (follower → leader), *disagg_step_up calls *wti_layered_drain_ingest_tables which runs drain workers to clear each ingest table. The drain worker fails with WT_ROLLBACK: conflict between concurrent operations when trying to clear the ingest table, causing the entire step-up to abort:

\_\_layered\_drain\_worker\_run, 831: Failed to clear ingest table "file:T00001.wt\_ingest": WT\_ROLLBACK: conflict between concurrent operations
\_\_disagg\_step\_up, 1061: Failed to drain ingest tables: WT\_ROLLBACK: conflict between concurrent operations
\_\_wti\_disagg\_conn\_config, 1263: Failed to step up to the leader role: WT\_ROLLBACK: conflict between concurrent operations

On the arm64 variant, the same error surfaces differently: the drain thread's work function returns an error, causing *thread_run to call *wt_panic_func ("Unrecoverable utility thread error"), which panics the engine. The release-stress-tests variant sees the panic when the test harness subsequently calls reconfigure(role=leader) and gets WT_PANIC back.

Evergreen Task / Link

Patch: https://spruce.corp.mongodb.com/version/6a4869ca2246a4000771a45e/tasks

Specific failing tasks:

• [ubuntu2004-release-stress-tests / format-stress-test-disagg-switch-2|https://spruce.corp.mongodb.com/task/wiredtiger_ubuntu2004_release_stress_tests_format_stress_test_disagg_switch_2_patch_6e21a082e8c3e08bbdab7b0c1fe6d250864d9b0a_6a4869ca2246a4000771a45e_26_07_04_02_03_02]
• [ubuntu2004-stress-tests-arm64 / format-stress-test-disagg-switch-2|https://spruce.corp.mongodb.com/task/wiredtiger_ubuntu2004_stress_tests_arm64_format_stress_test_disagg_switch_2_patch_6e21a082e8c3e08bbdab7b0c1fe6d250864d9b0a_6a4869ca2246a4000771a45e_26_07_04_02_03_02]

Logs & Stack Trace

Relevant log lines from the failing run (ubuntu2004-release-stress-tests):

\[Follower\] Picked up checkpoint \(metadata=\[metadata\_lsn=876831,...\],timestamp=0x1af01ce\)
\[WT\_VERB\_LAYERED\]\[WARNING\]: Picking up the same checkpoint again: metadata LSN = 1181058
\[WT\_VERB\_DEFAULT\]\[ERROR\]: \_\_layered\_drain\_worker\_run, 831: Failed to clear ingest table "file:T00001.wt\_ingest": WT\_ROLLBACK: conflict between concurrent operations
\[WT\_VERB\_DEFAULT\]\[ERROR\]: \_\_disagg\_step\_up, 1061: Failed to drain ingest tables: WT\_ROLLBACK: conflict between concurrent operations
\[WT\_VERB\_DEFAULT\]\[ERROR\]: \_\_wti\_disagg\_conn\_config, 1263: Failed to step up to the leader role: WT\_ROLLBACK: conflict between concurrent operations

Stack trace from arm64 variant (drain thread panics with unrecoverable error):

#2  \_\_wt\_abort \(session=0x11afffc95000\) at os\_abort.c:32
#3  *wt\_panic\_func \(error=\-31800, func="*thread\_run", line=47, "Unrecoverable utility thread error"\) at err.c:633
#4  \_\_thread\_run \(arg=0x11afef9db360\) at thread\_group.c:47

Concurrent eviction thread in the same process was stuck reconciling a dirty page during the step-up:

#15 \_\_wt\_reconcile at rec\_write.c:127
#16 \_\_evict\_reconcile at evict\_page.c:1334
#17 \_\_wt\_evict at evict\_page.c:490
#18 \_\_wti\_evict\_page at evict\_dispatch.c:254
#19 \_\_wti\_evict\_lru\_pages at evict\_queue.c:140

The conflict is likely caused by the concurrent eviction/reconciliation writing to the same ingest table that the drain worker is trying to clear during step-up.