-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
When sort or classic's group spill to disk, they will also spill to disk the last batch when they are done:
And in the PR for TEXT_OR stage.
This makes reading the results pretty straight-forward, but we pay the price of spilling an extra chunk, which can pretty big and even if it is small, we pay a lot of conversion overheads.
However sort merge iterator accepts a span of abstract iterators:
So it should work perfectly well if we pass a bunch of file iterators as well as an in-memory iterators for the last batch that was not spilled to disk.
It may complicate the code a little bit, but should come with some performance improvements. Especially for the cases where the stage spilled only once or twice.
- depends on
-
SERVER-99587 Use Sorter instead of SorterFile and iterators when spilling
-
- Needs Scheduling
-