Found while working on WT-18422 (prepared transactions across async step-down).

Summary

Assertion abort in __clayered_select_current (src/cursor/cur_layered.c) during a layered-cursor walk:

__clayered_select_current: WiredTiger assertion failed:
  'F_ISSET(c_current, WT_CURSTD_KEY_INT) || __clayered_ingest_prepare_blocked(op, c_current)'

__clayered_select_current
__clayered_advance_positioned
__clayered_iterate_constituents
__clayered_iterate_int
__clayered_iterate
__clayered_prev / __clayered_next
read_op -> nextprev -> table_op   (test/format ops thread)

Fires on both next and prev walks.

Root cause

The layered-cursor iteration code assumes that a constituent cursor which is positioned but carries no internal key can only ever be the *ingest* cursor, blocked on a prepared conflict. It states the underlying invariant at cur_layered.c:1141:

/*
 * If the old cursor has a position, copy it to the newly opened cursor. Prepared updates are
 * always ignored on the stable cursor, making it safe to check the WT_CURSTD_KEY_INT flag.
 */

Three helpers depend on it:

• __clayered_any_constituent_positioned (1846): "the stable cursor is positioned when it carries an internal key; the ingest cursor keeps a page reference through a prepared conflict that clears its key, so its reference is the reliable signal".
• __clayered_ingest_prepare_blocked (1857): only ever matches c_current == op->ingest.
• __clayered_select_current (1894): the failing assert, which tolerates a keyless cursor only via the helper above.

That invariant holds when prepared content lives exclusively on the ingest table, which is the case on develop – and is why develop forbids prepared transactions while the step-down timestamp is set.

A prepared update on the stable table breaks it. On a prepare conflict a btree cursor keeps its page reference while its key is not set, by design (src/btree/bt_curprev.c:790, matching code in bt_curnext.c):

case WT_PREPARE_CONFLICT:
    /*
     * If prepare conflict occurs, cursor should not be reset unless they have bounds and were
     * being initially positioned, as the current cursor position will be reused in case of a
     * retry from user.
     */

Nothing about that is ingest-specific. So once the stable table can hold a prepared update, the stable cursor can become positioned-but-keyless, and every one of the three helpers above misclassifies it.

Runtime confirmation

A temporary diagnostic added immediately before the assert (removed again afterwards) printed, on the failing walk:

WT18717: c_current=STABLE stable_key=0 stable_ref=0xe64c619939c0
         ingest_key=1 ingest_ref=0xe64c6061fd20
         clayered_flags=0x5 (ACTIVE|ITERATE_PREV) iface_key=0

c_current is the stable cursor with stable_key=0 and a non-NULL stable_ref – positioned but keyless, exactly the prepare-blocked signature the code attributes to ingest alone. The ingest cursor is healthy (ingest_key=1). An earlier core dump at the same assert showed the same shape (stable_cursor->flags with WT_CURSTD_KEY_INT clear, ingest_cursor->flags with it set).

Why prepared transactions are required

ops.prepare=1 is required to reproduce; with only that knob flipped to 0, 10 runs completed cleanly. This follows directly from the root cause: prepared updates are the only way to get a cursor into the positioned-but-keyless state, and a prepared update must be on the stable table for the stable cursor to reach it.

WT-18422 puts prepared updates on the stable table deliberately. Its straddler resolution clones a still-prepared update onto ingest and leaves the original in place on stable (src/txn/txn.c): "the original stays linked into the stable page, still prepared, until it is resolved in place below". That is what makes the scenario reachable.

preserve_prepared is not required: reproduced with preserve_prepared=0 on run 6 of 8.

Ruled out

An earlier hypothesis that mirrored writes (disagg.stepdown_write_mirroring) disturb the stable cursor's position via _clayered_put_both / _clayered_put_constituent is wrong. It predicted a repro with ops.prepare=0 (10/10 clean, falsified) and predicted a cleared stable position, whereas the diagnostic shows the stable cursor still holding its page reference. Mirroring puts committed content on stable, which never clears a cursor's key.

How to reproduce

On the WT-18422 branch (wt-18422-prepared-stepdown-straddler), no source modifications needed:

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DHAVE_DIAGNOSTIC=1 -DENABLE_PYTHON=1
cmake --build build
cd build/test/format
ulimit -c unlimited
for i in $(seq 20); do
    rm -rf RUNDIR$i && mkdir -p RUNDIR$i
    timeout 240 ./t -h RUNDIR$i -c /path/to/CONFIG > /tmp/run$i.log 2>&1
    if grep -q __clayered_select_current /tmp/run$i.log; then
        echo "reproduced on run $i"; break
    fi
    echo "run $i: no repro"
done

Intermittent, roughly 1 run in 2 with the attached CONFIG (which carries the seeds), each run about 80 seconds. A reproducing run exits 134 and logs the assertion; the core lands in the run directory. Non-reproducing runs end with t: successful run completed.

Relevant configuration beyond ops.prepare=1: disagg.mode=switch, disagg.stepdown_async=1, disagg.stepdown_write_mirroring=1, precise_checkpoint=1, runs.threads=7, runs.source=layered, runs.type=row-store, stress.disagg_stable_dhandle_delay=1. Which of these beyond ops.prepare are strictly necessary has not been narrowed down; mirroring in particular has not been shown to be required now that the mirrored-put hypothesis is ruled out.

Note on the earlier develop experiment

develop_enabling_deletions.patch and develop_repro.log are attached from an earlier attempt that deleted develop's two prepare prohibitions plus the test/format gate and reproduced the assert with no WT-18422 code present. Given the root cause above, that result is expected and not evidence of an independent develop defect: deleting those asserts is another way to put prepared updates on the stable table, which is the same violation. They are kept only as history.

No standalone Python reproducer yet

Notes from the attempts, to save the next person time:

• Concurrent walkers plus mirrored writers plus prepared writers produced 5673 prepare conflicts over 834k walk steps without hitting the assert. Those conflicts were on the ingest cursor, which is the handled case; the hard part is getting a prepared update onto stable and a walk to trip over it.
• A mid-walk write through the same cursor is not a valid approach: that legitimately resets the cursor and restarts the walk (verified identical on a plain non-layered table:). The write has to come from a different cursor/session.
• Announcing a step-down without completing it makes verify fail with EBUSY at teardown; that is a test artifact, not this bug.

Fix direction

Two options, both on the WT-18422 side rather than in develop's iteration code:

1. Teach the iteration layer that the stable cursor can also be prepare-blocked: generalise _clayered_ingest_prepare_blocked to either constituent, update _clayered_any_constituent_positioned to use the page reference for stable too, and revise the stale comment at cur_layered.c:1141. This makes the invariant match the new reality.
2. Keep prepared updates off the stable table entirely, so the existing invariant continues to hold.

Option 1 is the smaller change but widens the contract the iteration code has to honour, so it needs care around __clayered_advance_positioned, which currently drives only the ingest cursor when re-checking a blocked key.