Query limit in stream processing

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc1
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • Fully Compatible
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      SERVER-130817 changed the default of internalQueryMaxMemoryUsageBytesPerOperation from effectively unbounded (LLONG_MAX) to max(1GB, 20% of available memory). This is an operation-wide cap enforced across all memory-tracked stages/expressions of a single operation.

      Streams inherits this cap because the stream-processor ExpressionContext is built with an OperationContext. However, streams already has its own memory management: per-operator → per-processor ChunkedMemoryAggregator → node-level ConcurrentMemoryAggregator, enforced by KillAllMemoryUsageMonitor (tracked + RSS-based), with checkpoint recovery.

      So we do not want to enforce an operation-wide memory limit on stream processing, because streams governs total/accumulated memory through its own aggregator and node-level monitor. We do want, though, to keep a bound on individual expression evaluation. To do this, streams should use the default standalone per-expression tracker (capped by internalQueryMaxSingleExpressionMemoryUsageBytes) without propagating usage to the operation-wide OperationMemoryUsageTracker.

            Assignee:
            Projjal Chanda
            Reporter:
            Foteini Alvanaki
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: