HashAggBaseStage state memory usage extrapolated from single record

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      HashAggBaseStage::checkMemoryUsageAndSpillIfNecessary() (used by both HashAgg and BlockHashAgg) samples a single row at each memory checkpoint and extrapolates the hash table from it. This works well if the rows have similar size but they have uneven size the spilling decision depends on the order of the data rather than the data itself.

      This should be updated to be similar to WindowStage, which keeps a running mean over sampled sizes instead of just the latest one.

            Assignee:
            Unassigned
            Reporter:
            Foteini Alvanaki
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: