[SERVER-18927] Bound all memory usage by the agg pipeline Created: 11/Jun/15  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Charlie Page Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: usability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Query Execution
Participants:

 Description   

Aggregation should bound total memory usage for each pipeline, not like each individual stage. Using multiple stages that can each use up to 100MBs can result in overall very large memory usage.

Original Description:
The agg framework should bound all memory usage, not just the sort stage. Using very large amounts of data in $group stage, for instance, can result in 26G of non-mapped virtual memory for a single agg job.



 Comments   
Comment by Charlie Swanson [ 27/Nov/17 ]

asya there are some strange cases in which we can use more than 100 or even 200 MB of memory at a time, even though each stage is limited to 100MB. These include, but may not be limited to:

  • Using a $facet stage with multiple memory-intensive stages in different pipelines.
  • Using more-expressive $lookup to generate multiple memory-intensive stages at the same time.
  • Using $graphLookup multiple times in the same pipeline (I haven't verified, but I think you could get arbitrary memory usage since each one will keep a cache capped at 100MB).

Does that answer your question? I haven't tried to reproduce this issue, but I would believe you if this was resolved at some point, and the $group stage in particular will never use more than 100MB. However, I think that there are some cases for which this feature/enhancement would be valuable.

Perhaps we can convert the description of this ticket to only mention cases like the above that we know exist?

Comment by Asya Kamsky [ 23/Nov/17 ]

charlie.swanson I'm not sure this ticket is valid, since aggregation does limit memory usage in group as well as in sort to 100MBs.

I'm not able to verify this and this was originally reported on 2.6.x and was resolved by adding an index for leading $match stage, suggesting that maybe the issue was due to massive collection scan and not to aggregation memory use in particular.

Suggest we close as invalid unless some smoking gun can be found.

Generated at Thu Feb 08 03:49:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.