[CLASSIC] AndHashStage should spill if allowDiskUse is specified and memory usage has exceeded some limit

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      AndHashStage uses an unordered map to store the records from the first child. It tracks the memory this consumes and compares it against a limit defined in a variable inside the cpp file. We will convert this to a query knob and spill the data when the memory exceeds this limit instead of spilling.

       

      Additionally the unordered set it uses to track records already seen should be replaced by a RecordIdDeduplicator and spilled if necessary in order for the memory consumption to drop below the memory usage limit..

            Assignee:
            Unassigned
            Reporter:
            Foteini Alvanaki
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: