• Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      A single large $group spill record can take longer than the interrupt-check interval to serialize or deserialize. SERVER-136021 checks between records and merge partials; it does not bound work inside a record. SERVER-136213 adds classic spill-sort/$push checks and removes an SBE recovered-array deep copy, but large-record spans remain. These can delay killOp/maxTimeMS handling and produce interrupt-check delinquency.

      Evidence

      Scratch phase instrumentation on an ARM host, optimized builds using PR #65062 source at 0d7105f6354. One hot-key $push over 450,000 documents with 512-byte strings, 100MiB group budget, 512MiB push cap, allowDiskUse:true, and a trailing $project returning array length. ExecutionStats explain verified engines and real spills. Three measured executions followed setup explain.

      Phase in changed build Mean Maximum
      Classic spill writing/finalization, including accumulator extraction 45.01ms 87.80ms
      Classic next-partial read/deserialization 26.52ms 48.78ms
      Classic releasing/replacing previous partial 2.66ms 2.70ms
      SBE recovered-record deserialization 21.98ms 29.43ms

      Classic push merging itself took about 6ms mean / 9.6ms maximum; SBE transfer plus merge about 1.1ms mean / 1.4ms maximum. Those merge loops do not explain the remaining large-record spans in this shape.

      Write timing includes serialization, compression/checksum, and file/storage writing. Read timing includes reading/decompression and Value deserialization. Subcomponent attribution is still needed; these figures do not establish that storage waits are uninterruptible. These are instrumented sequential dev-host measurements, not throughput comparisons.

      Scope

      1. Split write/read timings into serialization/deserialization, compression/decompression, and storage access. Inspect classic and SBE paths and interrupt-check boundaries.
      2. Add amortized interrupt/progress checks within CPU loops that dominate long spans, or boundary checks where shorter sections combine into an overdue interval.
      3. Pass checking from the owning stage while preserving context-free serializer callers. Audit affected callers and exception safety before modifying shared helpers.
      4. Preserve spill format, accumulator order, memory accounting, and results. Do not mask active serialization through accounting exclusions or change delinquency thresholds.

      Repro

      Run the hot-key shape above in classic and SBE separately. Set overdueInterruptCheckSamplingRate=1.0; retain the default 30ms threshold. Verify usedDisk/spills and actual engine; observe matching aggregate/getMore slow logs and phase traces.

      Local DSI sources: workloads/query-execution/delinquent-queries/src/delinquent_queries_workload/group_repros.py in /home/ubuntu/dsi-interrupt-delinquency-repro. Cases: ClassicPushMerge and SbePushMerge. Workload additions are currently uncommitted and must be included explicitly in CI. Notes: workloads/query-execution/delinquent-queries/GROUP_PHASE_RESULTS.md in that worktree.

      Acceptance criteria

      • Deterministic cancellation tests cover the identified serialization/deserialization loops and exception cleanup.
      • Spill round-trip tests cover nested arrays/objects and large payloads without format or result changes.
      • Matched before/after repros show reduced check-free spans in targeted phases, distinguishing storage waits and unrelated output work.
      • Run sys-perf for hot-path changes before merge.

      Related

      • SERVER-136021 / PR #64882: between-record/partial checks and teardown exclusions.
      • SERVER-136213 / PR #65062: spill sorting, classic push checks, SBE array ownership transfer.

      Final result materialization and bulk teardown are outside this ticket's scope.

            Assignee:
            Unassigned
            Reporter:
            Kyle Burgess
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: