[SERVER-38761] Explore options for batch size when gathering shard results Created: 22/Dec/18  Updated: 04/Oct/23

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Justin Seyster Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: query-44-grooming
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-43657 Investigate whether there are situati... Backlog
Assigned Teams:
Query Execution
Participants:
Linked BF Score: 0

 Description   

Some of our benchmarks appear to be sensitive to the batch size that mongos uses to get documents from documents from cluster cursors. Currently, we use large batches, but we may want to consider an approach that adapts to more workloads.

Some possibilities:
1) Use a small batch size when the pipeline has a $limit that puts an upper bound on the number of documents that the mongos will need (perhaps upper_bound / num_shards, which would allow for one optimally sized batch from each shard in the best case).
2) Use small batches for the first few getMore commands and then ramp up batch size as execution continues.
3) Investigate if always using a small batch size is potentially a better trade-off than always using a large batch size.


Generated at Thu Feb 08 04:49:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.