Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-98103

Add spill API in SBE engine

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution

      Add a new virtual public method

      virtual void forceSpill() {
          // call forceSpill for all its children
      };
      

      in PlanStage class.

      Each method that implements spilling will override the forceSpill method to spill all its data to disk before calling forceSpill on if its parent.

      virtual void forceSpill() {
          spill();
          PlanStage::forceSpill();
      };
      

      If a method does not spill, the default forceSpill implementation will be used.

      During this task we should make sure that all classic stages that spill also maintain the following metrics

      • bool usedDisk : Set to true when the stage has spilled.
      • uint64_t spills : The number of times the stage spilled.
      • uint64_t spilledBytes : The size, in bytes, of the memory released with spilling.
      • uint64_t spilledDataStorageSize : The size, in bytes, of disk space used for spilling.

      We should also verify that they release all memory and disk when closed.

      The stages that can spill are:

      • BlockHashAggStage
      • HashAggStage
      • SortStage
      • WindowStage
      • HashLookupStage
      • HashLookupUnwindStage
      • UniqueStage
      • SpoolEagerProducerStage

            Assignee:
            Unassigned Unassigned
            Reporter:
            foteini.alvanaki@mongodb.com Foteini Alvanaki
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: