[SERVER-27829] Reduce performance impact of DocumentSourceCursor batching Created: 26/Jan/17 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Asya Kamsky | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | QFB, performance, query-44-grooming | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Query Execution
|
||||||||||||
| Sprint: | Query 2017-05-29 | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
On a large collection with index on "orderdate" compare find and equivalent aggregate first batches:
The main performance difference seems to come from keysExamined which is 101 in case of find and equal to I'm not sure what in the second case (there are 2406 distinct orderdate values, collection size is 1500000 and this filter reduces it to 1034289 documents). |
| Comments |
| Comment by Asya Kamsky [ 27/Jan/17 ] | |
|
It looks at the keys for first 4MBs worth of documents, vs find 101 keys. Note it caused aggregation to take 11ms vs 0ms for find. | |
| Comment by David Storch [ 27/Jan/17 ] | |
|
As part of the work to make find and agg share performance properties, we should look into getting rid of batching inside DocumentSourceCursor. I don't think it would be hard to make pipelines like this fully streaming. | |
| Comment by Charlie Swanson [ 27/Jan/17 ] | |
|
This is because the cursor stage of the aggregation will internally batch results. The current batch size is 4MB, but was recently reduced from 16MB in
Of course this is a tradeoff, we use a large batch size to hide the overhead of dropping and re-acquiring the lock. A query will hold the lock until it has filled up its first batch to return to the user, but an aggregation will only hold a lock during the cursor stage. If we only returned one document at a time from the cursor stage, we would have to re-acquire and drop the lock each time a result is requested from the cursor, which would add a lot of overhead. The internal buffering/batching is meant to mitigate this overhead. If you are only interested in time to first 101, you could add a $limit after the $sort, and it should be passed through to the query layer. You wouldn't be able to ask for more results from that cursor of course. |