Summary

The diagnostic assert added in commit WT-18419 correctly detected an application transaction still holding writes when async disaggregated step-down begins its final transition. The offending operation is a layered cursor reserve() issued by test/format, even though the worker is in the write-pause window and its selected operation is READ.

Configuration

The failing task ran the following effective format configuration:

disagg.enabled=1
disagg.layered=1
disagg.mode=switch
disagg.stepdown_async=1
runs.rows=1800
runs.tables=1
runs.threads=26
runs.timer=1
ops.reserve=9
ops.prepare=0
ops.truncate=0
ops.random_cursor=0
ops.verify=1
random.data_seed=84377
random.extra_seed=16394912

The command line from the core is:

./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch disagg.stepdown_async=1 ops.prepare=0 ops.truncate=0 ops.random_cursor=0 runs.rows=1000:3000 runs.tables=1:3 runs.timer=1

Steps to reproduce the analysis

1. Download the Complete task artifacts archive.
2. Extract the following files without needing to unpack the source tree:

   cmake_build/test/format/dump_t.934229.core
   cmake_build/test/format/t
   cmake_build/libwiredtiger.so.12.0.0
   cmake_build/dump_t.934229.stacktrace.txt
   cmake_build/test/format/RUNDIR/CONFIG
   

1. Open the core with the matching ARM64 executable and shared library. The local GDB needs the WiredTiger shared-library symbols loaded at the core's mapped address:

   gdb -q cmake_build/test/format/t cmake_build/test/format/dump_t.934229.core
   (gdb) add-symbol-file cmake_build/libwiredtiger.so.12.0.0 0xffff9e200000
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->id
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->mod_count
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->time_point
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->stepdown_ts_set
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->disagg_role_leader
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->mod[0].btree->dhandle->name
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->mod[0].type
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->mod[1].btree->dhandle->name
   (gdb) p ((WT_SESSION_IMPL *)0xffff9d6af940)->txn->mod[1].type
   

Expected observations:

• The assert fires at src/conn/conn_layered.c:1343 in __disagg_assert_no_active_writes_callback.
• The violating session is session ID 40, corresponding to format worker 23.
• txn->mod_count is 2 and the transaction ID is 210236.
• Both transaction operations are WT_UPDATE_RESERVE.
• The two btrees are file:T00001.wt_stable and file:T00001.wt_ingest.
• The transaction has stepdown_ts_set=true and disagg_role_leader=true.

The captured stack trace shows the assert running on the step-down thread:

__disagg_assert_no_active_writes_callback
__wt_session_array_walk
__disagg_assert_no_active_writes
__disagg_step_down_int
__disagg_step_down
__wti_disagg_conn_config
__wti_conn_reconfig
__conn_reconfigure("disaggregated=(role=follower)")
disagg_async_stepdown

Root cause

The worker loop samples pause_writes and forces the main operation selection to READ while the pause is set. However, table_op() performs its optional reserve operation before the main operation switch. The reserve condition only checks for an active snapshot transaction and ops.reserve, and does not check pause_writes.

Therefore, a paused worker can execute this sequence:

1. Start an explicit snapshot transaction.
2. Enter table_op() with the eventual operation selected as READ.
3. Execute the pre-operation reserve() path because ops.reserve=9.
4. The layered cursor mirrors the reservation into both stable and ingest.
5. Continue with the READ operation while intxn remains true.

The pause acknowledgment protocol assumes that a worker acknowledges only after its transaction is no longer in flight. That assumption is valid for the normal write path, but the reserve path creates a write before the pause-only READ restriction is applied. The core's two WT_UPDATE_RESERVE records are the direct evidence of this write.

Proposed fix

Pass the sampled pause state into table_op() and suppress the optional reserve block while paused:

static int
table_op(TINFO *tinfo, bool intxn, iso_level_t iso_level, thread_op op, bool pause_writes)

if (!pause_writes && intxn && iso_level == ISOLATION_SNAPSHOT &&
tinfo->ignore_prepare == false &&
mmrand(&tinfo->data_rnd, 0, 100) < GV(OPS_RESERVE))
{ ... }

Update all table_op() callers to pass the current pause_writes value. This preserves the intended behavior: existing transactions drain normally, but no new reserve write can be created once the worker write pause is raised.