ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      SERVER-137477 excludes SBE root close() and aggregation-stage dispose() from overdue interrupt-check accounting. Large frees also occur during execution. An interrupt check inside a container's bulk destruction cannot avoid completing that destruction, so attributing only that span to missed interrupt checks conflates required cleanup with interruptible execution. Do not exempt the surrounding spill, merge, or phase-transition work: interrupt checks before and after the free can still stop later work promptly.

      Measured non-teardown frees

      Temporary CLOCK_THREAD_CPUTIME_ID instrumentation around individual frees in an optimized aarch64 build measured the following scaled workloads at the default 30ms interrupt threshold. The instrumentation was removed after measurement. Slow-query logs on the same commands recorded overdue checks where noted, but do not prove that every overdue millisecond came from a free (especially during spill I/O).

      • src/mongo/db/exec/sbe/stages/block_hashagg.cpp: BlockHashAggStage::getNext() clears _ht when the final output block is produced, before executor close. Clearing 400,000 groups took 45-48ms of thread CPU; the final getMore logged one overdue check in each of three runs. With 120,000 groups, the clear took about 8-9ms and the final getMore logged none in five runs.
      • src/mongo/db/pipeline/group_processor_base.cpp: GroupProcessorBase::reset() replaces _groups between streaming-group batches, called from StreamingGroupStage::readyNextBatch() in src/mongo/db/exec/agg/streaming_group_stage.cpp. Resetting 200,000 groups took 107-120ms of thread CPU; each of three commands logged an overdue check (77-90ms excess).
      • src/mongo/db/exec/sbe/stages/hybrid_hash_join.cpp: HybridHashJoin::finishProbe() clears _ht when switching from probing to spill processing, even without spilled partitions. With 300,000 build keys, the clear took 83-84ms of thread CPU; three commands logged one, one, and two overdue checks (63-65ms total excess). Explain confirmed a join-optimizer HASH_JOIN_EMBEDDING / SBE hj plan, not EQ_LOOKUP.
      • src/mongo/db/exec/sbe/stages/hybrid_hash_join.cpp: HybridHashJoin::reset() clears _ht on reopen or close. In an optimized join-optimizer HASH_JOIN_EMBEDDING / SBE hj repro, clearing 450,000 rows took 97-104ms of thread CPU. With the guard disabled, four sampled queries recorded 75-85ms overdue; with the guard enabled, four sampled queries recorded no overdue interval. Exclude only the mandatory hash-table free, not spill-file cleanup or phase bookkeeping.
      • src/mongo/db/exec/sbe/stages/hashagg_base.cpp: HashAggBaseStage::spill() clears ht after writing partial groups and checks for interrupts afterward. With a 32MiB threshold, clearing 671,089 keys took 90-91ms of thread CPU; three spill commands logged 3, 3, and 5 overdue checks. Serialization and I/O in those commands also contribute to check gaps, so the clear's contribution needs an isolated before/after accounting test. With a 1MiB threshold, each 20,972-key clear took about 0.9-1.2ms even though spill commands still logged overdue checks.
      • src/mongo/db/exec/sbe/stages/lookup_hash_table.cpp: LookupHashTable::forceSpill() writes buffered inner values to disk, clears the buffer, writes hash-table indices to disk, then resets the in-memory hash table. The two frees were measured separately in optimized builds: resetting 450,000 in-memory keys took 47ms of thread CPU. Clearing 700,000 owned 512-byte values took 41ms, while clearing 450,000 inline integer values took 0.35ms. Exclude only each free; keep the intervening writes and serialization interruptible. The lookup timings establish that the frees can exceed the threshold, but do not establish a sampled query-level before/after.
      • HybridHashJoin::nextSpilledJoinCursor(): The previous partition's hash table held about 1,170-1,770 rows in tested optimized spill workloads. Individual clears took 42-578 microseconds of thread CPU, well below the 30ms threshold. Keep this clear accounted; do not add an exclusion for the entire partition transition. An overdue interval in a spill command is not evidence that this small clear caused it.

      Facet tee-buffer frees

      • src/mongo/db/pipeline/tee_buffer.cpp: TeeBuffer::loadNextBatch() clears the previous batch with _buffer.clear() before loading the next one. The SPM-4422 investigation used perf and temporal localization to attribute a free-side burst to this clear; reducing the facet buffer size eliminated the observed delinquency.
      • src/mongo/db/pipeline/tee_buffer.h: TeeBuffer::dispose() clears _buffer when the last consumer stops. A local $facet with only a $limit: 1 branch logged approximately 661-670ms overdue with the default 100MB buffer; reducing the buffer to 1MB eliminated that interval. The limit stage disposes its source after its first result, reaching this last-consumer clear. A subsequent scaled optimized repro timed this individual clear at 38-42ms for about 839,000 documents; sampled queries recorded 10-15ms overdue before the narrow guard and none with it. These scaled measurements do not attribute the entire earlier 661-670ms interval to this clear.

      Both paths free memory that must still be released after an interrupt. Exclude only the clear's destruction time from overdue interrupt-check accounting, not the subsequent batch loading or source disposal. Keep visibility into cleanup latency.

      Scope and acceptance criteria

      • Evaluate each listed free separately, including both TeeBuffer::loadNextBatch() and the last-consumer TeeBuffer::dispose() clear. Pause overdue accounting only across non-interruptible destruction where warranted, not entire getNext(), open(), spill(), forceSpill(), batch-loading, or source-disposal calls. Preserve checks before/after frees and cancellation, deadlines, memory accounting, output, and spill behavior. Leave the measured sub-threshold HybridHashJoin::nextSpilledJoinCursor() clear accounted.
      • Add focused tests distinguishing time spent freeing from preceding execution and subsequent work, including a cancellation that arrives before a free. Verify the effect on overdue counters with a mock clock and compare optimized workloads with sampling enabled. For tee-buffer paths, test both next-batch replacement and last-consumer disposal; confirm that overdue work preceding each clear remains recorded while time inside the clear is excluded, and that an interrupt stops further work after mandatory cleanup. Report paths too small or already covered rather than adding blanket exemptions.
      • Keep CPU-intensive spill serialization/deserialization interruptible; large-record gaps in those loops are tracked by SERVER-137887. Run sys-perf for any hot-path changes.

      Measurements were on scaled local repros, not an estimate of production prevalence. The cleanup timings directly establish that the retained frees can exceed the threshold; the spill case needs further causal isolation.

            Assignee:
            Kyle Burgess
            Reporter:
            Kyle Burgess
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: