Summary

In disaggregated mode on a follower, a role transition marks the shared history store's stable dhandle outdated and the sweep server closes it. The close path takes the dhandle write lock and then waits for the eviction pass lock inside __wt_evict_file_exclusive_on. The eviction server holds the pass lock for its whole pass and, at pass start, blocks on a read lock of the same dhandle while reopening its cached shared-HS cursor. This is an AB-BA deadlock between the eviction pass lock and the dhandle rwlock. The evict_walk_lock the sweep already holds then blocks every other close or drop in the process.

Observed downstream in MongoDB as BF-46986: on a stepped-down secondary, a replicated dropIdent of an index build's internal table queued on the walk lock, the oplog applier's batch never completed, and replication froze until the test's awaitReplication timed out after 600s.

Root cause

Stack traces from the hang-analyzer core of the stuck secondary (mongod, disagg follower):

Sweep server, holding the dhandle write lock and the walk lock, waiting for the pass lock:

__lll_lock_wait -> pthread_mutex_lock -> __wt_spin_lock_track
__wt_evict_file_exclusive_on        (waiting on evict_pass_lock)
__wt_conn_dhandle_close
__sweep_close_dhandle_locked
__sweep_expire_one
__sweep_expire -> __sweep_server
dhandle: file:WiredTigerSharedHS.wt_stable (outdated=true, timeofdeath=0,
  rwlock: writer active, readers_queued=1)

Eviction server, holding the pass lock across the whole pass, waiting for a read lock on the same dhandle while reopening its cached shared-HS cursor:

__wt_cond_wait -> __wt_readlock -> __session_dhandle_readlock
__wt_session_lock_dhandle           (dhandle: file:WiredTigerSharedHS.wt_stable)
__curfile_reopen -> __wt_cursor_cache_get -> __wt_curhs_get_cached
__evict_update_work -> __evict_pass -> __evict_server -> __evict_thread_run

Mutex __owner fields read from the core confirm the cycle: the pass lock is owned by the eviction server, the walk lock is owned by the sweep server, and the shared-HS stable dhandle is write-locked by the sweep server with the eviction server queued as a reader.

The lock ordering violation

Two orderings collide:

• Eviction server: pass lock -> dhandle read lock, in the cache-hit reopen done by __wt_curhs_get_cached.
• Sweep server: dhandle write lock, from WT_WITH_DHANDLE_WRITE_LOCK_NOWAIT in __sweep_expire_one -> pass lock, from WTI_WITH_PASS_LOCK in __wt_evict_file_exclusive_on.

The safety comment in __wt_curhs_get_cached (from WT-18924) says the caller is safe because "sweep must close the btree before freeing it, which waits for that lock". That only covers use-after-free of the btree. It misses that the sweep takes the dhandle write lock before waiting for the pass lock, while the eviction server acquires the dhandle read lock under the pass lock. Each side holds what the other needs.

How the shared-HS dhandle became sweepable

The WT-7053 exclusion (WT_IS_HS skip in the sweep's time-of-death marking) only covers the idle-close path. The outdated-discard path in __sweep_expire closes outdated dhandles immediately with no HS exclusion. On a follower, a step-down or checkpoint pickup marks the shared-HS stable dhandle outdated, so the next sweep pass closes it even though the eviction server still reopens cached cursors against it. This matches the open scope of WT-18356, which tracks eliminating the live shared-HS handle on followers, including the eviction pre-cache path.