Summary

On a sustained, IO-bound `w:majority` write workload, a single secondary in a 3-node replica set can fall progressively behind, fail to recover, and go into maintenance mode (`REPL` log id 21126, "Too stale to catch up"). The other secondary keeps pace, so the majority commit point stays healthy and flow control never engages.

Two questions:
1. Why does a secondary applying the oplog do more write IO than the primary that originated the writes? In healthy runs of this workload, FTDC shows both secondaries issuing more "write requests issued" than the primary.
2. Once a secondary slips, why does it struggle to recover, even when the workload has quieter periods that should give it time to catch up?

Evidence

Earlier reproductions: the lagging secondary shows ballooning system memory dirty, sustained high disk queue length, elevated system cpu iowait, a rising wiredTiger perf file system write latency histogram total (msecs) rate, and cache multi-block reconciliation blocked whilst checkpoint is running right before the lag increases. It seems like a cache/eviction problem tipping into foreground eviction.

A later reproduction looks different: system cpu iowait and the WT write-latency histogram are not materially higher, but the lagging node shows longer checkpoints, higher reconciliation maximum milliseconds, smaller disk average write request size, and more log force write operations skipped. So there may be more than one path into the same state.

An earlier Storage Engines review of the FTDC couldn't isolate the cause from the captures we had: history-store content moves were small (~500ms of total reconciliation), and there was no oversized hot page at eviction. The conclusion was that a repeatable reproducer is needed to investigate properly. That reproducer is being built under PERF-8925, which this depends on.

Why this is separate from other flow control work (SERVER-92061)

Flow control (FC) throttles the primary to bound replication lag, but it doesn't engage here. FC paces against the median apply rate across members, and in a 3-node PSS set the median follows the faster secondary, so a single slow secondary never triggers throttling. Flow control is also the wrong layer for the root cause: it can slow the primary, but it can't explain or prevent why one secondary does more write IO and then spirals once it lags. Treat flow control (SERVER-92061) as a possible safety net for the symptom and this ticket as the root cause.

Proposed investigation

1. Use the reproducer from the linked ticket to capture FTDC with whatever extra instrumentation Storage Engines needs.
2. Explain the secondary IO amplification: why an oplog-applying secondary issues more write IO than the primary, and whether application-thread foreground eviction is the mechanism that tips it over.
3. Determine whether the two FTDC signatures are one mechanism or two.
4. Decide whether a secondary-side apply throttle tied to cache pressure is warranted (slow the apply rate before the node goes into foreground eviction), versus relying on the primary-side flow control safety net in SERVER-92061.

Acceptance criteria

• A repeatable reproducer is linked and usable.
• The secondary IO amplification has an explanation backed by FTDC.
• A recorded decision on whether a server change is warranted. If yes, a follow-up implementation ticket is filed. If no, the reasoning is written down.