Issue Summary

A btree cursor that walks forward, blocks on a prepared update, and then reverses direction can panic in the diagnostic key-order check. It happens when the reversal itself hits a prepare conflict (a second prepared update on the last returned key) and the next reversal attempt, after the conflict is resolved, returns that key. The check reports "returned key key4 then key key4" and the process aborts.

WT-18968 fixed the same panic for the case where the first reversal call returns a key. It does not cover the case where the first reversal call conflicts, because the call clears the retry flag that tells the next return to restart the check.

Reproduction

# table: key2, key4, key6, key8 committed at ts 10; stable ts 10.
# Session A prepares an update on key6 at ts 12.
cursor = session.open_cursor(uri)
session.begin_transaction('read_timestamp=14')
cursor.next()   # key2
cursor.next()   # key4
cursor.next()   # WT_PREPARE_CONFLICT on key6
# Session B prepares an update on key4 at ts 30.
session.commit_transaction()
session.begin_transaction('read_timestamp=35')
cursor.prev()   # WT_PREPARE_CONFLICT on key4
# Session B rolls back.
cursor.prev()   # panics: WT_CURSOR.prev out-of-order returns: returned key key4 then key key4

[WT_VERB_OUT_OF_ORDER][ERROR]: WT_CURSOR.prev out-of-order returns: returned key key4 then key key4
__cursor_key_order_check_row: found key out-of-order returns: Invalid argument

Reproduced on a plain table, on a diagnostic build, on the tip that already contains the WT-18968 fix. No layered table is involved.

Context

• src/btree/bt_curprev.c and src/btree/bt_curnext.c reset the key-order check on a successful return only when WT_CBT_ITERATE_RETRY_NEXT or WT_CBT_ITERATE_RETRY_PREV is set on entry.
• Both functions clear the opposite retry flag at the end of every call, including a call that returns WT_PREPARE_CONFLICT. The conflicted reversal therefore consumes the flag, and the later successful return finds it cleared.
• Only diagnostic builds run the check, so the effect is a spurious panic rather than wrong results.
• A layered cursor reaches this through its stable btree cursor when a walk blocks and then reverses twice across prepare conflicts.

Proposed Solution

• Keep the reversal marker until a call returns a key, or reset the key-order check at the point of the conflict, so the next successful return restarts the check.
• Add a test next to the WT-18968 test with the sequence above, for both directions.

Definition of Done

• The reproduction above passes on a diagnostic build in both directions.
• The existing WT-18968 test still passes.