-
Type: Task
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
While spilling in SBE group and lookup stages, we insert record by record to the spill store (each followed by a commit), instead of inserting a batch of records together and reducing the number of IO. Note the classic document sources which uses filesystem for spilling does batching when writing to files.
I tested with a small patch that does batching when spilling in hash_group.cpp and the throughput of the $group queries that spill increases upto 25% (higher improvement when there are many groups and the individual groups are small leading to more spill records)