-
Type: Task
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
SpoolLazyProducerStage in addition to returning the results to the caller, it also saves them in buffer. Currently there is no limit on the amount of memory that can be used by this buffer.
To set the maximum allowed memory, we will use the internalQueryMaxSpoolDiskUsageBytes query knob that is already used by the classic SpoolStage.
The stage will spill to disk when the data structure exceeds the maximum memory allowed.
The spilling should be implemented in a method
void spill(uint64_t maximumMemoryUsage)
that will spill until the memory used by the stage is at most maximumMemoryUsage. The method should track the following metrics
- bool usedDisk : Set to true when the stage has spilled.
- uint64_t spills : The number of times the stage spilled.
- uint64_t spilledBytes : The size, in bytes, of the memory released with spilling.
- uint64_t spilledDataStorageSize : The size, in bytes, of disk space used for spilling.
We will track those metrics using the SpoolStats struct. The metrics should be reported in serverStatus and in explain execution stats.
Before spilling, the stage should make sure that there is enough disk space for spilling. This can be done using ensureSufficientDiskSpaceForSpilling and uassertStatusOK.
SpoolLazyProducerStage does not read the buffer. Thus, there is no need to add a method that will retrieve the spilled data.
The stage should release all memory and disk when it is closed.
- has to be done after
-
SERVER-98093 SpoolConsumerStage should be able to retrieve spilled data
- Backlog