[SERVER-31421] MongoDB crashes with "Out of memory" Created: 05/Oct/17 Updated: 16/Nov/21 Resolved: 12/Feb/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.4.6 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Tanveer Madan Marate | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Participants: |
| Description |
|
MonogDB crashes with "Out of memory"
|
| Comments |
| Comment by Ramon Fernandez Marina [ 12/Feb/18 ] | ||||||||||||||||||
|
This ticket has been dormant for a while so I'm going to close it. For further MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources. Regards, | ||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 26/Oct/17 ] | ||||||||||||||||||
|
Hi Tanveer, The situation you describe:
is normal and expected. The resident memory is actual physical memory in use by mongod, whereas virtual memory is just address space, only the resident portion of which is backed by real memory. The reason this situation arises has to do with the way the allocator manages memory. Every installation of mongod will show this characteristic, and I am not aware of any circumstances where a large virtual memory causes problems - it is the resident memory that is the potentially scarce resource. The previous diagnostic data ended a few seconds before the out of memory error crash (because the diagnostic data is only flushed to disk every few seconds, for performance reasons), so we couldn't be sure that mongod hadn't allocated a lot of memory in that few seconds. The new set of mongod data has the same characteristic, however the syslog shows that some other applications were also showing out-of-memory errors:
Crucially, these errors occurred a few minutes before mongod crashed at 14:21, and the mongod diagnostic data shows that it is not using excessive resident memory at that point. Given the above, I don't see any evidence of a bug in mongod that is causing of the out-of-memory errors. However unfortunately I also can't identify what is the cause. If you have the Linux OOM killer enabled, possibly it will be activated the next time this event occurs, and if that is the case it will collect detailed memory statistics and write them to syslog. If you are able to collect that information we could take a look to get a better idea of what's going on. Thanks, | ||||||||||||||||||
| Comment by Tanveer Madan Marate [ 24/Oct/17 ] | ||||||||||||||||||
|
Hi Bruce, I have uploaded the required files. Hope this clarification helps. I am trying to understand the memory usage ti restrict the use of memory we limited the WT cache to 120GB but even after that the mongod process seems to be using high memory Thanks, | ||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 24/Oct/17 ] | ||||||||||||||||||
|
Thanks Tanveer. Can you please also attach the mongod log file showing the crash, and clarify what you mean when you said it crashed with the same symptoms? Also can you please attach the syslog file covering the latest crash? Thanks, | ||||||||||||||||||
| Comment by Tanveer Madan Marate [ 24/Oct/17 ] | ||||||||||||||||||
|
Hi Bruce, Thanks for the response !!! Thanks, | ||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 24/Oct/17 ] | ||||||||||||||||||
|
Hi Tanveer, Can you please archive the diagnostic.data directory and attach to this ticket, along with the log file showing the crash? I would like to check whether all the circumstances are the same. Also, if you still have the syslog file showing the OOM details for the original incident can you attach that? It will have further information about memory usage. Thanks, | ||||||||||||||||||
| Comment by Tanveer Madan Marate [ 23/Oct/17 ] | ||||||||||||||||||
|
Hi Bruce, The database again crashed today with identical symptoms, exception being OOM was not invoked this time Thanks, | ||||||||||||||||||
| Comment by Tanveer Madan Marate [ 05/Oct/17 ] | ||||||||||||||||||
|
Hi Bruce, Thanks for your reply !!! Below is the limits for the mongod process (Note I restarted the mongod after the crash)
Thanks, | ||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 05/Oct/17 ] | ||||||||||||||||||
|
Hi Tanveer, Thanks for providing the diagnostic.data and logs. I'm not finding an indication that mongod was using an excessive amount of memory. The last data point recorded in diagnostic.data was at 10-03 17:08:01 UTC, about 7 seconds before the OOM. At this point mongod was using 150 GB resident memory, well short of the 256 GB of memory on the machine. It was using about 230 GB of virtual memory (78 GB of which was unmapped, leaving only the 150 GB resident that I mentioned above). Can you check the ulimits for mongod to make sure that is not limiting the virtual memory size of mongod? The best way to do this is
which will check the limits in effect for the actual running mongod process. If you post the results here we can check whether process memory limits are the cause of the issue. Also, can you confirm that there is nothing else running on the machine consuming memory, for example another mongod process? If that is the case you will need to adjust the cache limit used by mongod correspondingly. Thanks, |