When spilling in SBE group and lookup stages, batch records when inserting to the spill store.

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      While spilling in SBE group and lookup stages, we insert record by record to the spill store (each followed by a commit), instead of inserting a batch of records together and reducing the number of IO. Note the classic document sources which uses filesystem for spilling does batching when writing to files.

      I tested with a small patch that does batching when spilling in hash_group.cpp and the throughput of the $group queries that spill increases upto 25% (higher improvement when there are many groups and the individual groups are small leading to more spill records) 

              Assignee:
              Unassigned
              Reporter:
              Projjal Chanda
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: