Investigate changes in SPM-3701: Query Improvements for Memory Management

XMLWordPrintableJSON

    • Type: Investigation
    • Resolution: Won't Do
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Tools and Replicator

      Original Downstream Change Summary

      This project has created the `releaseMemory` command. The `releaseMemory` command is an internal tool designed to manage memory usage by forcing cursors to spill their data to disk. This command accepts an array of `cursorIds`. Consequently, subsequent `getMore` operations for these cursors will experience degraded performance due to the need to retrieve data from disk. Although the command attempts to keep the cursors alive, an issue during the spilling process may terminate them, resulting in a "cursor not found" error on subsequent `getMore` calls.

      Description of Linked Ticket

      Epic Summary

      Summary

      For cases where we have a large number of relatively idle, "heavy" (memory intensive) cursors, we'd like to be able to reclaim some of the memory held by the query execution operators inside the cursor. Rather than focusing our efforts on developing the right policy for spilling, we'd like to introduce a new mechanism for some tuning process to manually ask cursors to reclaim memory by spilling some of their query execution state to disk. After brief discussion yesterday on the topic, this work is likely broken up into two pieces:

      • Introduce new API for PlanExecutor so that we can make a request to reclaim memory
      • Introduce a new server command which allows requesting that some executors reclaim memory (likely by accepting cursor ids as a parameter)

      When implementing this, we should also consider any new metrics we might add to validate our implementation in production. 

      Motivation

      OOO Analysis

      Documentation

      Product Description
      Scope
      Technical Design
      Docs Update

            Assignee:
            Unassigned
            Reporter:
            Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: