WT-9041 has added a section of code which waits for transactions to complete, __wt_txn_checkpoint_cannot_start so it can be sure that all the relevant transactions have committed.
However if that transaction attempts to take a SCHEMA_LOCK the system can dead-lock.
The order of operations is:
- Checkpoint prepare is called within a WT_WITH_SCHEMA_LOCK while a live transaction exists in the system
- Checkpoint waits for that transaction to complete with while(__wt_txn_checkpoint_cannot_start)
- That transaction attempts to open a cursor on a new dhandle with __wt_session_get_dhandle which takes a WT_WITH_SCHEMA_LOCK.
- Both threads wait indefinitely.