Background

WT-18362 stopped checkpoint's connection-wide gather walk from reopening a duplicate handle when it finds one that sweep has marked DEAD mid-walk. That removed the majority of the checkpoint-prepare stalls reported on HELP-98165 (the __wt_evict_file_exclusive_on call inside the reopen path).

Reproducing the same workload against develop (which has WT-18362) shows a residual stall that WT-18362 does not address: live thread-stack sampling during a stall shows the checkpoint thread blocked in _wt_readlock / wt_session_lock_dhandle, while the sweep thread, at the same instant, is inside wt_evict_file_exclusive_on called from wt_conn_dhandle_close (via _sweep_expire_one).

Mechanism

_sweep_expire_one takes the dhandle's rwlock in write mode via WT_WITH_DHANDLE_WRITE_LOCK_NOWAIT, then calls wt_conn_dhandle_close(mark_dead=true) — including the slow wt_evict_file_exclusive_on handshake — while still holding that write lock. Checkpoint's gather walk, reaching the same handle at roughly the same time, blocks trying to take the read side of the same lock in _wt_session_lock_dhandle, and has no way to know in advance whether the write-lock holder is a quick transition or a slow one.

This is a different, uninstrumented lock from the connection-wide handle-list lock (conn->dhandle_lock), which already has wait-time statistics. The per-handle rwlock does not — this is presumably the gap WT-18354 is asking about.

Proposed fix (prototyped, patch attached)

Restructure the connection-wide walk in __wt_conn_btree_apply (only for WT_SESSION_IS_CHECKPOINT) into two passes:

1. First pass, in list order: probe each open handle with the existing non-blocking __wt_try_readlock (already used elsewhere, e.g. evict_lru.c, txn.c). Uncontended handles proceed exactly as today. A handle whose lock is currently held is deferred (an extra WT_DHANDLE_ACQUIRE keeps it alive) rather than blocked on inline.
2. Second pass, after the walk: resolve every deferred handle with the existing, unmodified blocking acquire — so the final decision (apply/skip) is always made from the authoritative locked state, identical to today's correctness guarantee. Nothing is ever skipped without that determination; only when the wait happens changes.

The rationale: the per-handle write-lock hold is inherently transient (open/close/special-op, never indefinite), so by the time the deferred pass revisits a handle, many will have already released — turning what would have been an inline wait into an instant grab.

Reproducer results (prototype, one run per configuration — see caveats)

Built mongodb-8.0 (unfixed), develop (WT-18362 only), and develop + this patch; ran an identical workload (15k tables, periodic fan-out touching all of them, a rotating dirty set, frequent checkpoints, short close_idle_time to force collisions). Comparing develop-with-WT-18362 against develop+patch, on a per-checkpoint basis (raw totals are not comparable between runs — the patch measurably increases overall checkpoint/fan-out throughput, so more work fits in the same wall-clock window):

The patch eliminated every stall ≥ 5s in this run and cut the worst case by ~3.5x, at the cost of a small increase in typical-case (median) overhead — consistent with the extra probe lock/unlock pair paid on every uncontended handle.

Caveats, stated plainly:

• This is one run per configuration at one parameter set — enough to trust the direction (and the 7-vs-0 severe-tail elimination is a strong signal), not enough to certify the exact percentages. An earlier attempt at this comparison, before a checkpoint-cadence artifact in the test driver was fixed, showed the opposite result; this version corrects that, but has not been replicated.
• Compiled with zero warnings and ran several hundred checkpoints across four separate runs with no crash or assertion, which is reasonable evidence of basic robustness, not a substitute for review.
• Not yet checked: whether any other caller reaching this same connection-wide walk with WT_SESSION_IS_CHECKPOINT set could see a handle busy for a reason other than a sweep close (e.g. a concurrent bulk-load or drop) where deferral behaves differently than the existing blocking wait.
• A second, related idea was considered — shrinking sweep's write-lock hold time itself by moving _wt_evict_file_exclusive_on outside of it — but _sweep_close_dhandle_locked on current develop has ingest-table/WT_BTREE_AWAITS_PUBLISH/timestamp bail-out logic that must run before deciding whether to actually close, so a blind hoist risks paying the eviction-disable cost on trees that bail out anyway. Not attempted; flagged here as a follow-on if this patch alone doesn't fully close the gap.

Patch attached (conn_dhandle_two_pass_defer.diff), against develop.

Related

• HELP-98165 — the customer ticket this investigation traces back to.
• WT-18362 — the fix this builds on top of (removes the majority of the stall; this ticket addresses what's left).
• WT-18354 — proposed dhandle close/sweep wait-time statistics; would make this residual visible in production without live stack sampling.