[SERVER-39909] mongod killed by oom-killer Created: 01/Mar/19  Updated: 06/May/19  Resolved: 06/May/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.12
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: PMB Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Hello,

I am trying to gain insight into an ongoing issue we've been having. Some configuration info:

  • MongoDB 3.2.12
  • centos-release-6-8.el6.centos.12.3.x86_64 running in a 64GB VMware VM

Two stand-alone instances of mongod run on this server: no sharding, no replication. There are several other memory consumers of memory on this server, mostly 5 or 6 Java programs only one of which consumes any significant memory (~16GB). And this one Java program is the main application that depends on MongoDB.

What we observe is everything working OK for 2-5 days and then the oom-killer decides to kill one of the mongods. This has happened 5-6 times over the last month. Typically, only 1 mongod is killed but there was at least one occasion where both were.

Output from 'sar' shows the same pattern again and again: consumption of RAM, sometimes fairly rapid, over the 2-5 days, followed by ~61GB of RAM in use for a day or two and then the oom-killer does its thing.

I should mention that I tried to constrain the WT cache size to 12GB for each of the two mongods. This seemed to prevent the oom-killer from firing, but our application became 'unresponsive'.

I should also mention that I've read a ton of MongoDB Jiras on this issue and, while I know that 3.2.12 is getting long in the tooth, many of the improvements in WT's memory management were supposedly, though not exclusively, in 3.2.10.

As to our application's 'access patterns', it's difficult to be precise but my sense is that it combines periodic bursts of write activity along with the occasional (human-driven) reading of a very large collection. (In this regard I am familiar with the Jiras that discuss MongoDB threads turning their attention to cache eviction rather than servicing application requests - but I believe that this issue was improved in 3.2.10).

In any event, I would be very grateful for some bright light on this ongoing and very frustrating problem. At a minimum, if I could upload the WT diagnostics and someone at MongoDB could run their internal visualizer against it - along with an analysis, that would be a good start.

Thank you for your help.

 

 

 



 Comments   
Comment by Danny Hatcher (Inactive) [ 25/Apr/19 ]

saultocsin are you still experiencing this issue?

Comment by Danny Hatcher (Inactive) [ 06/Mar/19 ]

Additionally, as you may have noticed through your testing, you will need to manually decrease the WT cache size from its default if you continue to run multiple mongod processes on the same server. The default currently uses a percentage of the total amount of RAM available on the box and does not account for other processes when doing so.

Comment by Danny Hatcher (Inactive) [ 06/Mar/19 ]

Hello,

I apologize; I should have been more clear regarding my suggestion of cgroups. There's an issue we've commonly seen in the past when a mongod process is co-located with other processes on a server, especially other mongod processes. If another process starts to cannibalize more memory on the system, say a different mongod receives a sharp increase in load, it can cause a cascading effect where your original mongod all of a sudden has less memory to allocate to itself. If you have the machine resources pre-allocated in cgroups, a rise in memory in one process has a much better chance of not killing other processes. However, as you said this will not prevent a single mongod from running out of memory so it was simply something to take into consideration moving forward.

In terms of the memory issue, upgrading would be your best bet. The primary reason I suggested going to the latest version of 3.4 instead of 3.6 or 4.0 was that there are less considerations for compatibility changes as compared to your current version of 3.2. If you feel comfortable upgrading even further, please do so! Whatever version you decide, please let me know if the upgrade resolves the issue or if you are still experiencing memory pressure. I can take a further look then.

Thanks,

Danny

Comment by PMB [ 06/Mar/19 ]

Hi Daniel,

Thank you for getting back to me about this and apologies for my delayed
reply.

I have considered, and even experimented with, Linux's overcommit tuning
knobs. But unless I am sorely mistaken, they don't really address the
issue. Rather, they shift the failure from oom-killer to the application
itself, i.e., a MongoDB malloc call might fail which I suspect will cause
the process's termination.

Can you explain how the use of cgroups might solve WT's memory problem?

Might I also ask why you recommend 3.4.19 rather than, say, 3.6 .11or a 4.X?

Thanks again.

On Fri, Mar 1, 2019 at 5:01 PM Daniel Hatcher (Jira) <jira@mongodb.org>

Comment by Danny Hatcher (Inactive) [ 01/Mar/19 ]

Hello,

I'm sorry to hear that you've been having these extended issues. Unfortunately, as you've pointed out 3.2.12 is an old version and has officially reached end-of-life status. There were some improvements that were made in 3.2.10 but improving WiredTiger performance was and continues to be an ongoing process. Additionally, 3.2.x lacks some diagnostic capability that future versions provide. Would it be possible for you to upgrade to 3.4.19? If you do, we can take a look at the status of your cluster that results.

As a note, we generally recommend using one mongod process per server with no major competition in system resources. If you must host everything on the same server, you may wish to try using a mechanism like cgroups to assign resources to your mongod processes.You may be able to avoid some OOM events if you don't have the mongod processes competing with each other for available RAM.

Thank you,

Danny

Generated at Thu Feb 08 04:53:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.