-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
A single large $group spill record can take longer than the interrupt-check interval to serialize or deserialize. SERVER-136021 checks between records and merge partials; it does not bound work inside a record. SERVER-136213 adds classic spill-sort/$push checks and removes an SBE recovered-array deep copy, but large-record spans remain. These can delay killOp/maxTimeMS handling and produce interrupt-check delinquency.
Evidence
Scratch phase instrumentation on an ARM host, optimized builds using PR #65062 source at 0d7105f6354. One hot-key $push over 450,000 documents with 512-byte strings, 100MiB group budget, 512MiB push cap, allowDiskUse:true, and a trailing $project returning array length. ExecutionStats explain verified engines and real spills. Three measured executions followed setup explain.
| Phase in changed build | Mean | Maximum |
|---|---|---|
| Classic spill writing/finalization, including accumulator extraction | 45.01ms | 87.80ms |
| Classic next-partial read/deserialization | 26.52ms | 48.78ms |
| Classic releasing/replacing previous partial | 2.66ms | 2.70ms |
| SBE recovered-record deserialization | 21.98ms | 29.43ms |
Classic push merging itself took about 6ms mean / 9.6ms maximum; SBE transfer plus merge about 1.1ms mean / 1.4ms maximum. Those merge loops do not explain the remaining large-record spans in this shape.
Write timing includes serialization, compression/checksum, and file/storage writing. Read timing includes reading/decompression and Value deserialization. Subcomponent attribution is still needed; these figures do not establish that storage waits are uninterruptible. These are instrumented sequential dev-host measurements, not throughput comparisons.
Scope
- Split write/read timings into serialization/deserialization, compression/decompression, and storage access. Inspect classic and SBE paths and interrupt-check boundaries.
- Add amortized interrupt/progress checks within CPU loops that dominate long spans, or boundary checks where shorter sections combine into an overdue interval.
- Pass checking from the owning stage while preserving context-free serializer callers. Audit affected callers and exception safety before modifying shared helpers.
- Preserve spill format, accumulator order, memory accounting, and results. Do not mask active serialization through accounting exclusions or change delinquency thresholds.
Repro
Run the hot-key shape above in classic and SBE separately. Set overdueInterruptCheckSamplingRate=1.0; retain the default 30ms threshold. Verify usedDisk/spills and actual engine; observe matching aggregate/getMore slow logs and phase traces.
Local DSI sources: workloads/query-execution/delinquent-queries/src/delinquent_queries_workload/group_repros.py in /home/ubuntu/dsi-interrupt-delinquency-repro. Cases: ClassicPushMerge and SbePushMerge. Workload additions are currently uncommitted and must be included explicitly in CI. Notes: workloads/query-execution/delinquent-queries/GROUP_PHASE_RESULTS.md in that worktree.
Acceptance criteria
- Deterministic cancellation tests cover the identified serialization/deserialization loops and exception cleanup.
- Spill round-trip tests cover nested arrays/objects and large payloads without format or result changes.
- Matched before/after repros show reduced check-free spans in targeted phases, distinguishing storage waits and unrelated output work.
- Run sys-perf for hot-path changes before merge.
Related
SERVER-136021/ PR #64882: between-record/partial checks and teardown exclusions.SERVER-136213/ PR #65062: spill sorting, classic push checks, SBE array ownership transfer.
Final result materialization and bulk teardown are outside this ticket's scope.
- is related to
-
SERVER-136213 Prevent interrupt delinquency inside $group spill merge iterations and the pre-spill sort
-
- Closed
-