[SERVER-47066] mongod leaks memory at ~4.8 MB/s when inserting documents at a high rate Created: 23/Mar/20 Updated: 27/Oct/23 Resolved: 05/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Andre M | Assignee: | Dmitry Agranat |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Backwards Compatibility: | Fully Compatible |
| Operating System: | ALL |
| Steps To Reproduce: |
|
| Participants: |
| Description |
|
Running the attached Mongo shell script with perfmon shows mongod's Private Bytes grow at 4.8 MB/s. The gap in the middle is when the test finished the first time. When it was then restarted, memory kept climbing.
Interestingly enough, if the test collection is dropped at the end of the test, then when the script is launched again, memory stays at the same level for a while. I didn't have the time to look closer into further behavior. If the collection is left alone and the script is restarted, memory keeps climbing. I see the same behavior in the application. Make sure to restart mongod when running it for the first time - it appears that it uses existing memory for a while and it is not as easy to spot the leak.
|
| Comments |
| Comment by Dmitry Agranat [ 05/Apr/20 ] |
|
Thank you for providing all the additional information, it was very useful. As suggested, I will go ahead and close this case. Regards, |
| Comment by Andre M [ 02/Apr/20 ] |
|
Just wanted to add for completeness, since I already have the data. Not changing any conclusions above, just hoping somebody finds this data useful. Looking at document insert rate, it appears that exhausting cache capacity does not change the document insert rate. This tells me that while it is useful as a buffer for disk I/O, these 14 GBs are not used much as a cache, since it's doing only inserts. Here's the chart for one client inserting documents (X is test duration in seconds and Y is docs/sec). Notice that the insert rate does not change when the 14 GB point is reached, which corresponds to the top perfmon chart above.
Here's the chart for 3 clients inserting documents (X is test average duration in seconds and Y is average docs/sec for all 3 clients). Notice that the insert rate changes only 20 minutes after reaching 14 GB, which means that something else is at play.
|
| Comment by Andre M [ 01/Apr/20 ] |
|
Thank you for the details, Dima. I want clarify the terminology I used colloquially, which may have caused some confusion. A memory leak can be called as such only if one ran the app in a memory consumption profiler, like VTune or AQtime, which would indicate that some memory was never released. I did not use any of those and the term "memory leak" was not correct in my narrative. I should have called the issue "mongod consumes memory at ~4.8 MB/s when inserting documents at a high rate". Sorry about that. Now that we are talking about memory consumption rate, I see that you did reproduce the same behavior in terms of memory climbing, except that in your case the rate is about 1.2 MB/s. My guess is that because it is much lower than the 4.8 MB/s I mentioned, you said that you could not reproduce the issue. Based on your observations I ran a few more tests and I can say that the growth rate is defined by speed of the test bed. It seems that my hardware is about 4 times faster than you used for the chart above, hence the difference in memory growth rates. Looking through the docs, I see that mongod should consume about half of RAM if inMemorySizeGB is not set. If it is indeed the mongod cache that consumes this memory, memory growth should stop after reaching that range, give or take. This perfmon chart shows a test that inserts 10 M documents and when it reaches about 14 GB, memory does flatten.
This perfmon shows 3 clients doing the same type of insert and mongod reaches 14 GB about twice as fast.
So, these tests do confirm that it is indeed some cache in mongod that consumes this memory. The thought of that did cross my mind when I was creating this issue, but because I specifically made this test only to insert data, the thought of the server caching data that will never be used didn't seem like a reasonable explanation for this memory growth pattern. In a real app I expect the same usage - the primary would mostly insert data and data would be queried in bulk mostly on the secondary. Anyway, please go ahead and close this issue. My apologies for the confusion - I suppose I'm grasping at straws because of the impact of Thanks for looking into this. |
| Comment by Dmitry Agranat [ 01/Apr/20 ] |
|
Thank you for providing the repro script. We tried to reproduce the issue you are reporting but based on the results, were not able to identify a memory leak. This plot visualize the execution of the provided script: Metrics explanation:
The same as in your example, two insert tests were executed (A-B and C-D). The gap in the middle that you've mentioned is marked by B-C. The increase in the resident memory "mem resident" is due to the increase in the allocated bytes "tcmalloc generic current_allocated_bytes" and that's in turn due to filling up the cache "wt cache fill ratio". The magnitude of the increase that we are seeing is roughly the same as you are seeing and it's the same rate of feeling the wt cache. In this repro we do not see a memory leak. Next step: We'll need to review the diagnostic.data from your repro in order to understand why your repro is behaving differently from ours. Thanks, |
| Comment by Andre M [ 24/Mar/20 ] |
|
Not quite sure why are you asking for diagnostics. I provided a 100% reproducible case for mongod v4.2.3 running on Windows. If you can reproduce the leak with the shell script I provided, there is no point gathering additional evidence. If for some inexplicable reason mongod is not leaking in your environment, then this bug won't do any better than |
| Comment by Dmitry Agranat [ 24/Mar/20 ] |
|
Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) covering the time of this test and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Thanks, |