[SERVER-18927] Bound all memory usage by the agg pipeline Created: 11/Jun/15 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Charlie Page | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | usability | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Assigned Teams: |
Query Execution
|
||||
| Participants: | |||||
| Description |
|
Aggregation should bound total memory usage for each pipeline, not like each individual stage. Using multiple stages that can each use up to 100MBs can result in overall very large memory usage. Original Description: |
| Comments |
| Comment by Charlie Swanson [ 27/Nov/17 ] |
|
asya there are some strange cases in which we can use more than 100 or even 200 MB of memory at a time, even though each stage is limited to 100MB. These include, but may not be limited to:
Does that answer your question? I haven't tried to reproduce this issue, but I would believe you if this was resolved at some point, and the $group stage in particular will never use more than 100MB. However, I think that there are some cases for which this feature/enhancement would be valuable. Perhaps we can convert the description of this ticket to only mention cases like the above that we know exist? |
| Comment by Asya Kamsky [ 23/Nov/17 ] |
|
charlie.swanson I'm not sure this ticket is valid, since aggregation does limit memory usage in group as well as in sort to 100MBs. I'm not able to verify this and this was originally reported on 2.6.x and was resolved by adding an index for leading $match stage, suggesting that maybe the issue was due to massive collection scan and not to aggregation memory use in particular. Suggest we close as invalid unless some smoking gun can be found. |