Summary

Query memory load shedding cannot shed operations that are waiting for an execution ticket. On the spill_to_disk availability workload this is the entire gap between the in-server shedder (arm E, 0.41 availability, crash-looping) and the mongotune killOp actor (arm D, 0.97 availability, zero failovers) as visible in the attached qmls-spill-to-disk-ab-report.html

Evidence

Analyzed 2026-09-23 from Atlas M50 A/B runs (base 472d159296b6, patches 6ab2d999 killOp / 6ab306ad static 30/70 shedder) in qmls-spill-to-disk-ab-report.html

During the memory storm, execution control shrinks the primary's read-ticket pool from 128 to ~10-14. Every aggregation is deprioritized and waits in the lowPriority read queue holding its 20-30 MB sort buffer.

In D, queue departures = admissions (160) + kills (142): about half of all ops leaving the queue were killed without ever getting a ticket back. In E, departures equal admissions exactly: the only exit is a ticket, memory-holding ops arrive at ~270/s against ~180 tickets/s, and the queue and heap grow until the kernel kills mongod.

Victim populations are identical (tracked p50 21RT / $setWindowFields), so this is not size calibration or eligibility.

Root cause

queryMemoryCheckLoadShedding is called only from checkpoints that require holding a ticket: PlanYieldPolicy::yieldOrInterrupt, the ExpressionContext interrupt checker, and the SBE no-yield check. A queued op cannot reach any of them.

The queued op also cannot check itself while waiting. Execution control uses OrderedTicketSemaphore (kPrioritizeFewestAdmissions), whose acquire does an untimed waitForConditionOrInterruptUntil: the thread wakes only when handed a permit or when markKilled notifies the baton. The yield-restore path enters via Locker::lockGlobal with Date_t::max(), so the 100 ms reacquireTicket loop is not involved. This is exactly why killOp works: the kill wakes the waiter immediately.

Secondary: the 1000 ms dt cap in the inline check under-credits an op that waited 5-20 s in the queue.

Non-options

• Hooking the shed check into the ticket wait: there is no polling loop to hook, adding a timed wake to OrderedTicketSemaphore touches permit-handoff code, and lock_manager cannot call memory_tracking because memory_tracking:query_memory_load_shedding already depends on shard_role/lock_manager (Bazel cycle).
• Relying on more/earlier checkpoints: execution control's ticket shrink is what starves them; any yield-based design fights that policy and does nothing in the EC2 thrash-stall case where nothing is admitted.

Recommended implementation

Sweep ticket-queued operations from the existing RSS-monitor PeriodicRunner job (queryMemoryRssMonitor, 100 ms), so the decision runs on a thread that is never blocked.

1. Gate each tick on the cheap above-low-mark check (PressureSnapshot::aboveLowMark from SERVER-132002) so healthy servers do no work.
2. Walk clients with Service::LockedClientsCursor. For each client with an OperationContext, keep only ops where:
   • the Locker client state is kQueuedReader or kQueuedWriter (atomic read), so the op is provably not holding a ticket;
   • isOperationQueryMemorySheddingEligible is true;
   • wasGlobalLockTakenForWrite is false (same exemption as the inline path);
   • CurOp::getInUseTrackedMemoryBytes() > 0 (atomic, published by the tracker).
3. Roll with rollForShed (SERVER-132002) using the sweep interval as maxDt and the monitor's own PRNG.
4. On a hit, call ServiceContext::killOperation(clientLock, opCtx, ErrorCodes::QueryMemoryLimitExceeded) while holding the client lock, then do the existing operationsShed / OTel bookkeeping and rate-limited log.

Restricting to the queued state partitions ownership: a ticket-holding op is decided by the inline check, a queued op by the sweeper, so hazard is never double-counted and the sweep skips running ops and non-query clients.

State hazard to resolve: LoadSheddingOpState::lastEvalTime is written by the op thread at its checkpoint and would now also be written by the sweeper. Either make it atomic or have the sweeper keep no per-op state and charge one sweep interval per pass.

Expected size: ~60-80 lines in query_memory_load_shedding.cpp, one atomic in the op-state struct, a serverStatus counter for queued sheds, no changes to locker or ticketing.

Optional cheap addition: a second inline check immediately after restoreTransactionResourcesToOperationContext in PlanYieldPolicy, with the dt cap lifted for that call, so an op dies as soon as it regains a ticket instead of after another work slice. Bounded by admissions/s, so not sufficient on its own.

Relationship to SERVER-132002 (idle-cursor shedding)

Same problem class: queries that can never reach the inline checkpoint. SERVER-132002 covers queries with no thread (idle cursors); this covers queries whose thread is blocked without a ticket. Coverage is disjoint: the idle-cursor sweep skips pinned cursors, and a queued op's cursor is pinned.

Reusable from that branch as-is: PressureSnapshot / samplePressure, rollForShed with caller-supplied maxDt, shared counters and OTel mirrors, the rate-limited log pattern, and the "decision returns bool, caller kills" entry-point style. Add a third public entry point beside queryMemoryShouldShedIdleQuery.

Divergences: cadence host (cursor monitor thread vs RSS monitor job), kill mechanism (dispose cursor vs killOperation), and the lastEvalTime race, which the idle path never had.

Acceptance

• Unit test: an eligible op in kQueuedReader state with tracked bytes above the low mark is killed with QueryMemoryLimitExceeded by the sweep; a ticket-holding op is not touched by the sweep.
• Re-run spill_to_disk_atlas (n=5) with static 30/70 or 70/85 plus the sweeper; target parity with arm D: availability > 0.9, zero failovers, lowPriority queue length bounded.
• Confirm sheds/s exceed lowPriority admissions/s during the storm (the signature that queued ops are being shed).