ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      Add an observe-only ("dry run") mode to the query-memory load-shedding feature (SERVER-130333) that evaluates the full shed decision but does not abort operations, recording what would have been shed instead.

      Motivation

      Load shedding's only action is aborting user queries with a retriable QueryMemoryLimitExceeded. It ships off by default and is intended to be driven by external policy (e.g. mongotune). Before enforcing on a real workload, an operator/policy needs to know how many and which operations would be shed at a given calibration, without impacting queries:

      • Calibration — the low/high water marks and queryMemoryLoadSheddingSizeReferenceBytes are workload-dependent; dry run lets them be tuned against real traffic first.
      • Rollout safety — "observe for a while, then enforce" is the standard low-risk path for a destructive control.
      • Policy input — mongotune can consume the projected shed rate to decide when to enforce.

      Proposed design

      • New server parameter queryMemoryLoadSheddingDryRun (bool, default false), orthogonal to the existing queryMemoryLoadSheddingLowMarkPercent enable. Off = -1 low mark; observe = low mark >= 0 and dryRun=true; enforce = dryRun=false.
      • In queryMemoryCheckLoadShed(), run the unchanged evaluation (RSS, pressure, size, dt/priming/throttle, probabilistic roll). At the point the enforce path calls markKilled() and returns the error, the dry-run path instead records a "would-shed" and returns Status::OK().
      • One-shot semantics: latch "would shed" on the operation's memory tracker so it is counted/logged at most once per op, keeping wouldShed comparable to totalShed (ops, not roll events).
      • Observability: a wouldShed counter (serverStatus + OTel, mirroring totalShed) and a rate-limited debug log identifying the op (namespace, redacted command, planSummary, opId).

      Testing

      • Unit: reuse the queryMemoryPressureOverride + queryMemoryLoadSheddingAlwaysShed failpoints; with dry-run on assert the op is NOT killed but wouldShed increments; with dry-run off the existing shed behavior holds.
      • jstest coverage that an over-mark eligible op completes in dry-run while wouldShed advances.

      Considerations

      • Adds one server parameter and one OTel metric name. metric_names.h is owned by server-networking-and-observability via CODEOWNERS, so N&O review is required for the new metric.
      • Feature remains off by default; dry run has no effect unless explicitly enabled.

            Assignee:
            Unassigned
            Reporter:
            Steve Tarzia
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: