-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
None
-
None
-
None
-
None
-
None
-
None
-
None
HashAggBaseStage::checkMemoryUsageAndSpillIfNecessary() (used by both HashAgg and BlockHashAgg) samples a single row at each memory checkpoint and extrapolates the hash table from it. This works well if the rows have similar size but they have uneven size the spilling decision depends on the order of the data rather than the data itself.
This should be updated to be similar to WindowStage, which keeps a running mean over sampled sizes instead of just the latest one.