The deadlock

A disaggregated leader step-down can deadlock with eviction threads writing to the shared history store:

• The step-down thread holds the checkpoint lock, the schema lock, and the handle list read lock while it walks the handle list. For each open disaggregated btree it disables eviction and waits for in-flight eviction to drain (spin on btree->evict_busy in __wt_evict_file_exclusive_on).
• An eviction worker holding evict_busy on such a btree is inside reconciliation, which must write updates to the shared history store. Opening an HS cursor acquires the handle list read lock.
• The handle list rwlock is writer-fair: when a writer is queued, new readers queue behind the writer (mtx_rw.c).
• The writer can be the sweep server closing an idle table, a drop, or an eviction worker whose HS pre-cache had to allocate a fresh dhandle. It queues behind the step-down's read lock.

Cycle: step-down waits for eviction to drain; eviction waits behind the queued writer; the writer waits for the step-down's read lock. No thread can proceed.

The lock ordering rule

The documented hierarchy (arch-locking-hierarchy.dox) covers checkpoint -> schema -> handle list, and the eviction-internal spinlocks, but does not cover eviction state. The rule that keeps this system deadlock-free is:

A thread holding eviction state (evict_busy) must never block on the handle list lock or the schema lock. Eviction state is a leaf.

The step-down deadlock violates it from both sides at once: step-down holds the handle list lock while waiting on evict_busy, and eviction holds evict_busy while blocking on the handle list lock.

History: fourth variant of the same deadlock

• WT-5785 / WT-5946 (2020): a session wanting exclusive access to a handle (drop/verify/close) held the handle list write lock while waiting for eviction to drain; eviction needed an HS cursor. Fixed by pre-caching an HS cursor in every session that can evict (PRs 5389, 5509, 6083). Left behind FIXME-WT-6037: "This isn't reasonable and needs a better fix."
• WT-7053 (2023): sweep could close the HS dhandle, invalidating the cached cursor and forcing eviction back onto the lock. Fixed by excluding HS dhandles from sweep (PR 9161).
• WT-14957 (2025): disagg checkpoint pickup marked the HS dhandle outdated, invalidating the cache; the eviction server under the pass lock needed the schema lock for a fresh HS open. Fixed by making the eviction walk use only the cached HS dhandle (PR 12139).

Each fix removed one specific way eviction could take a lock mid-eviction. The root cause was never removed.

What step-down does differently

Step-down does two new things that together defeat all three past fixes:

• It holds the handle list read lock across the entire walk, including every per-btree eviction drain. Past drainers (drop, sweep, verify) hold the write lock around a single file.
• It marks the shared HS dhandle WT_DHANDLE_OUTDATED mid-walk. The handle list is newest-first (TAILQ_INSERT_HEAD), so after any prior role change the recreated HS handle sits near the start and is marked early. Both cache layers reject OUTDATED handles on sight: the session dhandle cache discards them (session_dhandle.c:152) and cursor cache reopen fails WT_DHANDLE_CAN_REOPEN (dhandle.h:66). The 2020 mitigation is disabled at exactly the moment the drain depends on it, and the next HS open escalates to the handle list write lock to allocate a fresh dhandle.

Stack traces from the core (format test, ASan)

Step-down thread, spinning in the drain while holding the locks:

__wt_yield()                                   evict_exclusive.c:164
__wt_evict_file_exclusive_on
__disagg_mark_btrees_readonly_then_step_down   conn_layered.c:1273
__disagg_step_down_int                         conn_layered.c:1332
__disagg_step_down                             conn_layered.c:1447
__conn_reconfigure("disaggregated=(role=follower)")

Four eviction workers, each holding evict_busy, blocked opening the shared HS:

__wt_readlock                                  mtx_rw.c:223
__session_find_shared_dhandle                  session_dhandle.c:887
   ("file:WiredTigerSharedHS.wt_stable")
__wt_session_get_dhandle
__wt_curfile_open
__wt_open_cursor
__curhs_file_cursor_open                       cur_hs.c:62
__wt_curhs_open
__wti_rec_hs_insert_updates                    rec_hs.c:1187
__rec_hs_wrapup / __rec_write_wrapup
__wt_reconcile
__evict_reconcile
__wt_evict
__wti_evict_page                               evict_dispatch.c:254

The queued writer: an eviction worker's own HS pre-cache, escalated to the write lock because the OUTDATED HS dhandle had to be reallocated:

__wt_writelock                                 mtx_rw.c:441
__session_find_shared_dhandle                  session_dhandle.c:894
__wt_session_get_dhandle
...
__wt_curhs_cache                               cur_hs.c:166
__evict_thread_run                             evict_thread.c:83

Fix proposal and analysis in comments.