-
Type: Task
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
Add a new virtual public method
virtual void forceSpill() {
// call forceSpill for all its children
};
in PlanStage class.
Each method that implements spilling will override the forceSpill method to spill all its data to disk before calling forceSpill on if its parent.
virtual void forceSpill() { spill(); PlanStage::forceSpill(); };
If a method does not spill, the default forceSpill implementation will be used.
During this task we should make sure that all classic stages that spill also maintain the following metrics
- bool usedDisk : Set to true when the stage has spilled.
- uint64_t spills : The number of times the stage spilled.
- uint64_t spilledBytes : The size, in bytes, of the memory released with spilling.
- uint64_t spilledDataStorageSize : The size, in bytes, of disk space used for spilling.
We should also verify that they release all memory and disk when closed.
The stages that can spill are:
- BlockHashAggStage
- HashAggStage
- SortStage
- WindowStage
- HashLookupStage
- HashLookupUnwindStage
- UniqueStage
- SpoolEagerProducerStage