-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Summary
Add an observe-only ("dry run") mode to the query-memory load-shedding feature (SERVER-130333) that evaluates the full shed decision but does not abort operations, recording what would have been shed instead.
Motivation
Load shedding's only action is aborting user queries with a retriable QueryMemoryLimitExceeded. It ships off by default and is intended to be driven by external policy (e.g. mongotune). Before enforcing on a real workload, an operator/policy needs to know how many and which operations would be shed at a given calibration, without impacting queries:
- Calibration — the low/high water marks and queryMemoryLoadSheddingSizeReferenceBytes are workload-dependent; dry run lets them be tuned against real traffic first.
- Rollout safety — "observe for a while, then enforce" is the standard low-risk path for a destructive control.
- Policy input — mongotune can consume the projected shed rate to decide when to enforce.
Proposed design
- New server parameter queryMemoryLoadSheddingDryRun (bool, default false), orthogonal to the existing queryMemoryLoadSheddingLowMarkPercent enable. Off = -1 low mark; observe = low mark >= 0 and dryRun=true; enforce = dryRun=false.
- In queryMemoryCheckLoadShed(), run the unchanged evaluation (RSS, pressure, size, dt/priming/throttle, probabilistic roll). At the point the enforce path calls markKilled() and returns the error, the dry-run path instead records a "would-shed" and returns Status::OK().
- One-shot semantics: latch "would shed" on the operation's memory tracker so it is counted/logged at most once per op, keeping wouldShed comparable to totalShed (ops, not roll events).
- Observability: a wouldShed counter (serverStatus + OTel, mirroring totalShed) and a rate-limited debug log identifying the op (namespace, redacted command, planSummary, opId).
Testing
- Unit: reuse the queryMemoryPressureOverride + queryMemoryLoadSheddingAlwaysShed failpoints; with dry-run on assert the op is NOT killed but wouldShed increments; with dry-run off the existing shed behavior holds.
- jstest coverage that an over-mark eligible op completes in dry-run while wouldShed advances.
Considerations
- Adds one server parameter and one OTel metric name. metric_names.h is owned by server-networking-and-observability via CODEOWNERS, so N&O review is required for the new metric.
- Feature remains off by default; dry run has no effect unless explicitly enabled.
- is related to
-
SERVER-130333 Probabilistic query memory load shedding
-
- Closed
-