As part of WT-11060, the logic for txn_mod_compare has been modified to:

1. Explicitly sort operation types without keys (e.g. WT_TXN_OP_REF_DELETE and WT_TXN_OP_NONE)
2. Fix the original comparator logic that doesn't correctly handle sorting on btree->ids (more information available in WT-11060).

We have run the failing test/format config for ~2 days with our new sorting algorithm and the original issue in WT-11060 has not reproduced. The purpose of this ticket is to provide an additional layer of confidence in the sorting algorithm via a cpp unit test.

This unit test should aim to cover corner cases of modifications that don't have keys. We want to test the correctness of __txn_mod_compare by calling it through __wt_qsort_r from a unit test. This should require mocking WT_TXN_OP, and creating a fake modification array to pass into the sort, then validating that it has come back sorted.

Scope:

• Mock WT_TXN_OP if it isn't already
• Create a list of unit tests to run on __txn_mod_compare