ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      Long-running query spilling, spill-run consolidation, and recovery/merging of spilled state can execute inside a single stage/executor call while retaining locks or other execution resources. Where Global IS remains held, setUserWriteBlockMode may be unable to acquire Global X promptly. Queued Global X can also block new Global IS acquisitions and stall unrelated operations.

      Interrupt checks alone do not release locks. Existing interrupt-delinquency fixes improve cancellation responsiveness but do not establish that global operations can progress when no cancellation or deadline is pending.

      This ticket covers an audit of query spill-capable operators and remediation of confirmed resource-yield gaps. Sort is the confirmed production example and initial implementation target; the same defect has not been demonstrated in every other spill path. Use linked implementation subtasks/follow-ups where lifecycle contracts differ, rather than assuming a single callback is safe everywhere.

      Production evidence

      HELP-102042 reports four unindexed $match -> $sort -> $facet aggregations on 8.0.34 spilling approximately 1.7TB. Mongotune issued setUserWriteBlockMode at 08:43:11, but Global X acquisition waited approximately 891 seconds until 08:58:02, coinciding with completion of spill merging. Scanning yielded, but sort consolidation ran inside loadingDone() -> mergeSpills() while retaining surrounding Global IS.

      The 8.0 wiredTiger serverStatus omission and eviction-poll errors were secondary symptoms. Newer serverStatus behavior does not establish that the underlying lock-yield problem is fixed. Confirm affected supported versions separately.

      Scope: audit all query spilling mechanisms

      Inventory query spill users and classify each execution path, including:

      • Classic and SBE query sorts: memory-pressure flushing, intermediate run consolidation, and reading final merged output.
      • Classic pipeline $group and SBE scalar/block/compact hash aggregation: spill writing and recovery/partial merging.
      • Hash lookup/join spilling: buffered-value writes, key/index-vector updates, bulk forced flushing, and disk-backed probing.
      • Spillable document maps/deques and their consumers, including window/pipeline buffering and graph/traversal stages where applicable.
      • Other spill-capable classic/SBE operators identified by the inventory, such as spool/text-related paths. Do not treat this list as an exhaustive caller audit.

      For each mechanism record:

      1. Actual locks/acquisitions, tickets, snapshots, spill cursors, and storage transactions retained during long work. A private spill recovery unit does not itself release outer query locks.
      2. Existing full yield opportunities and maximum work between them; distinguish interrupt-only checks from releasing locks/resources.
      3. Whether spilling executes within a collection-locked pushed-down executor or outside collection acquisitions in a pipeline stage.
      4. Natural memory-pressure spilling versus explicit releaseMemory/forceSpill lifecycle, including already-saved plans.
      5. Safe resumable boundaries and required ownership/save/restore work, or evidence that no added lock yield is necessary because the path does not retain the blocking resources.

      Implement remediation for confirmed gaps, starting with sort spill consolidation. Use bounded incremental work or full executor yielding with audited save/restore contracts. A yield between entire runs/batches is insufficient if one run/batch can take minutes. Split materially different implementations into linked tickets with explicit ownership and acceptance criteria.

            Assignee:
            Kyle Burgess
            Reporter:
            Kyle Burgess
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: