[SERVER-18241] MongoDB 3.0 + WiredTiger process uses more cache memory than specified Created: 28/Apr/15  Updated: 28/Apr/15  Resolved: 28/Apr/15

Status: Closed
Project: Core Server
Component/s: Stability, WiredTiger
Affects Version/s: 3.0.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Adam Wróbel Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File serverStatus.json     Text File top_memory.txt    
Issue Links:
Duplicate
duplicates SERVER-17424 WiredTiger uses substantially more me... Closed
Operating System: Linux
Participants:

 Description   

I've migrated my write-heavy application to MongoDB 3 and WiredTiger. This helped a lot with some of the performance issues I had, but now I am experiencing some memory problems on my DB host. My setup is fairly simple: I have an EC2 instance with 30GB system memory dedicated to running MongoDB; I only have one mongod process; I don't use sharding or replica sets.

When I first started using v3 and WiredTiger I've set the cache size to something large, like 27GB. Considering that I'm not running anything else on the instance, I thought it was safe to leave 3GB for the linux kernel and non-cache related MongoDB objects. My mongod was killed by OOM event on the next day. Then again after another day.

I've changed the cacheSizeGB to 24, leaving 6GB for the linux kernel and non-cache related MongoDB objects. Many days later I am running out of memory again.

I've been running the same application on Mongo 2.x for months without restarting the DB process. I don't use noTimeout cursors. I only have 8 web worker processes and 10 delayed_job processes accessing the database. That puts a cap of about 30-40 cursors that I might have open at the same time. It's far fewer most of the time.

I'm not sure how I can help you debug. `top` reports 26g memory used by the mongod process or 28.8g virtual image size. That's far above what is allowed for cache. Attached is the `top -abcn1` output and serverStatus info.

Possibly related tickets: SERVER-17456, SERVER-17424.



 Comments   
Comment by Adam Wróbel [ 28/Apr/15 ]

Ok, I will change the cache to default (15G) and we'll see if the process can outgrow it by another 10G. I will let you know if it does. It will likely take a long time though. Accumulating 3G over the allowed size took many days.

In the attached serverStatus.json "bytes currently in the cache" is much lower than "maximum bytes configured" so it would suggest WiredTiger thinks it isn't even using all of the available cache.

I thought WT internal cache was supposed to replace system-level file caching. I think there was even a mention of possibly moving to a raw disk layout in the future (space not controlled by file system). Seems kind of redundant to keep the same information cached in memory twice – once on the application level by WT and then again as system cache of the underlaying data file. Reads should be served by WT cache or by disk access while writes can be buffered by system, but only for very small chunks. Is that not the case? I don't have any other processes running on the instance that would require a lot of file system cache.

AMI EC2 instances by default come without swap configured and this one is no exception. I find it a very efficient setup for applications which memory usage I can predict. OOM events are results of bugs either in my application or higher in the stack and should either be fixed or accommodated for. Swap leads to unreliable performance and doesn't remove the need to restart the leaking process, only slightly delays it. In either case system memory usage needs to be monitored.

Comment by Ramon Fernandez Marina [ 28/Apr/15 ]

amw, as you point out the symptoms are similar to those of SERVER-17424. While we don't have evidence of a leak yet, it is the case that the cache usage may grow higher than specified.

I'd recommend that you use the default cache setting (which will be half of available RAM) and monitor your system to see how much more over the limit is used. The goal is to find out if the used cache size grows without bound (which could indicate a bug or a leak) or if it grows above the limit but to a steady number (just a bug).

Note that your current setting may starve the linux kernel from memory for file cache, which will negatively impact performance, so lowering the cache size may actually be advantageous for your write-heavy application.

The other important aspect is the amount of swap available in your system. Having low or no swap will prevent the system from degrading gracefully and force the OOM killer to step in.

I am going to close this ticket as a duplicate of SERVER-17424, but if you have some additional information that's specific to this ticket please log it here. If this information reveals that these two tickets have a different cause we'll reopen this one and continue the investigation.

Regards,
Ramón.

Generated at Thu Feb 08 03:47:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.