[SERVER-53764] Support spilling to disk in sbe::HashAggStage Created: 13/Jan/21  Updated: 06/Dec/22  Resolved: 25/Feb/22

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: David Storch Assignee: Backlog - Query Execution
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-58436 Implement spilling HashAgg Closed
Related
is related to SERVER-52793 Collect resource consumption metrics ... Closed
Assigned Teams:
Query Execution
Participants:

 Description   

The slot-based execution (SBE) engine currently supports spilling to disk for large blocking sort operations, but we still need to extend the hash-based aggregation operation to be able to spill pieces of its hash table. The HashAggStage has limited uses in plans constructed for find commands, but when we start using SBE to execute entire aggregation pipelines, we may choose a plan that implements MQL $group using sbe::HashAggStage. When the application sets allowDiskUse to true, the system should spill to disk when its reaches its allotted memory for the hash-aggregation rather than failing the query.

Note that as part of this work, we need to extend SERVER-52793. Namely, the HashAggStage must report execution stats about how many keys it sorted (if we spill the hash table sorted by key) and about how many times it spilled data to disk.


Generated at Thu Feb 08 05:31:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.