Goal

Add a seeded, randomized differential test that runs the same cursor operations on a layered table and on a plain (btree) table holding the same logical content, and checks that every result matches. The plain table is the reference: a layered cursor should behave exactly like a btree cursor over the merged contents of its ingest and stable constituents.

Background

While working on WT-18717 (layered iteration with prepared updates on either constituent), a prototype harness compared random layered walks against a plain table in lockstep. It found a real bug (a stale iteration flag after a prepared conflict, which tripped an assertion in diagnostic builds and could use a misplaced alternate cursor in release builds) and several cases where layered iteration diverged from btree semantics after a prepared conflict. These were fixed in WT-18717, but only a handful of deterministic cases were kept as regression tests. A permanent randomized test would catch similar divergences in future changes.

Prototype

Each seed:

• Builds a random key layout split between stable and ingest (ingest overriding some stable keys), with one or two prepared transactions on either constituent.
• Mirrors the same content, prepared transactions and timestamps into a plain table in a separate, non-disaggregated connection.
• Runs ~40 random operations in one reader transaction on both tables: next/prev, the reader transaction's own inserts and removes through another cursor, and resolving (rolling back) prepared transactions.
• Compares each result: key and value, WT_PREPARE_CONFLICT, or WT_NOTFOUND. Stops at the first divergence and reports the seed and the full operation trace.

The prototype covered the leader step-down window (stable blocked on a prepared update, writes routed to ingest).

Scope

• Setups:
  • Leader in the step-down window, with and without write mirroring.
  • Follower, with prepared updates on ingest and stable read from a checkpoint.
  • Isolation levels allowed in each setup (snapshot; read-uncommitted on a follower).
• Operations: next, prev, search, search_near, reset, own insert/update/remove/modify, prepared commit and rollback, and read context changes where allowed.
• Seeds fixed and reproducible; a failure prints the seed and the trace so it can be replayed.
• Runs in the Python suite for a bounded seed count; a larger count can run as a long or nightly variant.

Known differences to account for

These are known, intentional or out-of-scope differences; the test should classify them explicitly rather than fail, and each should be removed from the allowance once fixed:

• Early conflict: when the current constituent blocks on a prepared update but the other constituent has an earlier key, the layered cursor reports the conflict where a btree cursor would return that key.
• Own insert on the current constituent during an alternate stall: after the alternate blocks, the transaction's own insert into the current constituent, between the last key returned and the alternate's blocked key, is not returned on reversal.
• search_near reads past the target: after a read context change the alternate is repositioned with search_near, which can fall back to the previous key and hit a prepared update a btree cursor would not read.

Notes

• A plain table in a diagnostic build previously aborted in the key-order check when reversing out of a prepared conflict after keys were inserted during the stall; WT-18717 fixed that check, so the reference side should be stable.
• Expectations should never be hand-written: the plain table's result is the oracle.