Limit how many expensive operations can be active and holding resources at the same time

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Workload Resilience
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      SPM-4263 introduces a low priority queue for expensive operations so that they are de-prioritized compared to high-priority/short-running operations, limiting how many of them can be running at the same time with a ticket pool.

      However it doesn't put a cap on the number of operations that can be "active" (i.e. have started execution and are holding onto some resources) which means we can easily consume a lot of resources (specially memory) if many of these expensive/long-running operations are admitted.

      Limiting the number of operations that can queue in the low priority queue has the undesirable effect of potentially killing operations that had already done a lot of work and where close to finish.

      This ticket proposes adding a concurrency limiter at admission time that will reject new expensive/long running queries before they consume significant resources.

            Assignee:
            Unassigned
            Reporter:
            Daniel Gomez Ferro
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: