[SERVER-18519] Using $out in aggregation appears to leak non-mapped virtual memory Created: 18/May/15 Updated: 06/Dec/22 Resolved: 23/Feb/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | 2.6.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Sigfrido Narvaez [X] | Assignee: | Backlog - Query Team (Inactive) |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Assigned Teams: |
Query
|
| Operating System: | ALL |
| Steps To Reproduce: |
https://mms.mongodb.com/host/chartReplSetHosts/541a0803e4b07a2f96a3518a/541a0cabe4b0539fc6ff68d4 |
| Participants: |
| Description |
|
Using the c# driver, we run 200K+ aggregation jobs on a nightly basis where each uses $out to create a temporary collection. These collections are short lived and dropped after their data is moved to a permanent collection. Every time our jobs run we notice an increase of about 0.5gb of non-mapped virtual memory, which never releases, and in turn requires a periodic node re-start to free-up memory. |
| Comments |
| Comment by Charlie Swanson [ 23/Feb/16 ] |
|
I'm closing this ticket as "Won't Fix". The problem doesn't appear to affect 3.0, 3.2, or master, and we will not be doing a targeted patch for the 2.6 branch. If this problem is observed on any version since 3.0, please feel free to re-open. |
| Comment by Sigfrido Narvaez [X] [ 15/Jul/15 ] |
|
Update: We are still using 2.6 and have released a change on our code base to only use $out when needed (if an aggregation exceeds the 16mb limit). This has stopped the memory leak. We will upgrade to 3.0.4 soon and will report back. Cheers, Sig |
| Comment by Ramon Fernandez Marina [ 17/Jun/15 ] |
|
SigNarvaez, further investigation shows that only 2.6 is affected by increased memory consumption when using $out. In 3.0 there's an apparent increase, but this increase it's for data structures used by the lock manager that are later released and collected for reuse every minute. Would it be possible for you to test your application against a 3.0.4 node and report back? Would be useful to confirm that 3.0 addressed fixes your use case. Thanks, |
| Comment by Sigfrido Narvaez [X] [ 04/Jun/15 ] |
|
Excellent! Thanks for looking into this. |
| Comment by Ramon Fernandez Marina [ 03/Jun/15 ] |
|
Hi SigNarvaez, nice meeting you today. This is to let you know that we've been able to reproduce the memory increase you observe and we're investigating. Credit goes to craiggwilson for finding a fast repro! |
| Comment by Sigfrido Narvaez [X] [ 27/May/15 ] |
|
Stats taken during peak time of aggregation jobs |
| Comment by Sigfrido Narvaez [X] [ 27/May/15 ] |
|
Aggregations ; All queries use this connection profile: , }; return new MongoClient(clientSettings); server stats attached. DB, collection names and IP's are masked. |
| Comment by Sam Kleinman (Inactive) [ 19/May/15 ] |
|
Hello, In cases of apparent memory leaks, we want to be sure to rule out a case where open cursors or connections are the ones consuming resources. You can see current operations using db.currentOp(); and the output of db.serverStatus().metrics.cursor in the mongo shell. Can you please post the output of those operations you can post them to this ticket so that we can look over them together? Additionally, can you check your application to see you're setting the noTimeout option, or setting a custom timeout for your queries in your code? This can allow us to detect the source of non-expiring cursors that are not otherwise accounted for. Cheers, |
| Comment by Sigfrido Narvaez [X] [ 18/May/15 ] |
|
The image "memory increase 2 - fix.jpg" shows that the increase stopped after we moved away from using $out. Other aggregations that didn't require $out did not cause an increase, this can be seen in the other images. |