Background tasks may not be properly exempted from load shedding

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Workload Resilience
    • WR Prioritized list
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Claude says:

      There is an asymmetry in load shedding exemption logic:

      • ordered_ticket_semaphore.cpp:26-33 sheds only if getLowAdmissions() == 0 && queue >= maxWaiters && !admCtx->isLoadShedExempt().
      • unordered_ticket_semaphore.cpp:58-67 sheds on previousWaiters >= _maxWaiters alone — no isLoadShedExempt() check, and also no getLowAdmissions() == 0 check.

      Git archaeology — it is an oversight, not a design choice

      • UnorderedTicketSemaphore (with its overflow shed) landed 2026-03-02 in SERVER-119347 "Generalize TicketHolder to support pluggable concurrency primitives".
      • isLoadShedExempt() was added 2026-03-17 by SERVER-121825 "Make background operations bypass max queue length check" (ff515e0ef1b) — which touched only ordered_ticket_semaphore.cpp. The unordered variant already existed and was simply not updated. isLoadShedExempt() has exactly one caller in the tree.

      Reachability — the missing check is on the hot path

      • ExecutionAdmissionContext::isLoadShedExempt() returns getTaskType() == TaskType::Background (index builds, range deletions, TTL deletions).
      • In execution_control_init.cpp, the normal-priority read/write TicketHolders use the default SemaphoreType::kCompeting → UnorderedTicketSemaphore. Only the two low-priority holders pass kPrioritizeFewestAdmissions → OrderedTicketSemaphore.
      • Background tasks land in the low-priority pool only if deprioritization is on, and executionControlDeprioritizationGate defaults to false (execution_control_parameters.idl:161-172), which blocks all deprioritization regardless of executionControlBackgroundTasksDeprioritization.

      So in the default configuration, Background ops queue on the normal-priority unordered semaphore and get AdmissionQueueOverflow-shed despite being exempt. SERVER-121825's guarantee is effectively dead code out of the box; it only takes effect once the deprioritization gate is enabled.

      Secondary gap: unordered also drops the getLowAdmissions() == 0 guard, so an op that has already been admitted at low priority before can be shed there but not under the ordered semaphore.

      Fix is a two-line addition at unordered_ticket_semaphore.cpp:61, mirroring the ordered condition — note the ordering subtlety: _waiters has already been incremented at that point (ON_BLOCK_EXIT handles the decrement), so an exempt op that skips the uassert correctly stays counted as a waiter. ordered_ticket_semaphore_test.cpp:396 has a ready-made pattern (an AdmissionContext subclass overriding isLoadShedExempt()) to clone into a new unordered_ticket_semaphore_test.cpp — that test file doesn't exist yet, so the behavior is entirely uncovered.

      However, daniel.gomezferro@mongodb.com noted that in practice we only load shed from the low priority queue (the ordered semaphore), so there is no incorrect shedding behavior right now.

            Assignee:
            Unassigned
            Reporter:
            Steve Tarzia
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: