[SERVER-1121] Crashes reported on OpenVZ Created: 14/May/10  Updated: 10/Dec/14  Resolved: 23/Sep/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Richard Kreuter (Inactive) Assignee: Richard Kreuter (Inactive)
Resolution: Won't Fix Votes: 28
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-8606 Remove to change frightening OpenVZ m... Closed
is related to SERVER-11050 DBClientCursor::init call() failed Ca... Closed
Participants:

 Description   

Folks report crashes running MongoDB on OpenVZ. See downthread here:

http://groups.google.com/group/mongodb-user/browse_thread/thread/82a09dd33f3d377f



 Comments   
Comment by Ian Whalen (Inactive) [ 23/Sep/13 ]

Resolving as "Won't Fix" because this is a known issue in older versions of RHEL (RHEL 5 and previous).

Comment by W L [ 25/May/12 ]

I have a try on an openVZ shared server yesterday. CentOS release 5.8 (Final) Linux xxx 2.6.18-028stab091.2 #1 SMP Fri Jun 3 00:02:40 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux. The server have 7GB memory with only 10GB storage.
{
"db" : "test",
"collections" : 3,
"objects" : 5061126,
"avgObjSize" : 671.6067136048381,
"dataSize" : 3399086200,
"storageSize" : 4172689408,
"numExtents" : 32,
"indexes" : 11,
"indexSize" : 1688172304,
"fileSize" : 8519680000,
"nsSizeMB" : 16,
"ok" : 1
}
I am doing top to monitor at the same time, during insert mongod never use more than 420MB memory. indexing does take a fair bit. it hit around 3GB when I create the last three indexes after all the insert is done. The system does reclaim memory though as I monitor it. It will reclaim more memory if I do alternate insert and indexing. like insert 10 row/index a key/insert 10 more row/index another key.
Unfortunately I do not have the capacity to shoot through the roof of the memory. But mongo will have no problem to run if it has enough memory.

Comment by Andrew Luzin [ 08/Feb/12 ]

First, make sure that the privvmpages parameter is set to unlimited or a huge value. The memory consumed by the container must be limited by physpages. As Rich says, do repeated document inserts and monitor the amount of memory allocated by MongoDB. It is going to grow until it occupies all free memory. This is normal. When it happens, the operating system will sync data to a disk and free some memory. If the solution works, /proc/user_beancounters failcount for privvmpages should be 0.
Don't forget, this solution works only in new OpenVZ (RHEL6 or higher).

Comment by Rich Thornett [ 07/Feb/12 ]

As you do repeated document inserts, the MongoDB memory footprint grows. In a healthy environment, when that footprint approaches some max threshold that a system can handle (based on total memory and other processes running), memory allocation will eventually flatline as the operating reclaims memory to keep MongoDB in check. On OpenVZ, memory is never reclaimed and the footprint grows unchecked until the machine becomes unstable. (On my server, this manifests with "cannot allocate memory" errors when you attempt to access or do anything on the box.)

So create a few indexes, start adding a lot of data and watch your memory grow. That should give you an answer.

Comment by Richard Kreuter (Inactive) [ 07/Feb/12 ]

Not really exactly, but if you're seeing MongoDB running stably on OpenVZ for more than a few minutes with any non-zero write volume, that might suggest the problem has been resolved in recent versions of OpenVZ.

Comment by Jorge Costa [ 04/Feb/12 ]

I'm evaluating mongodb 2.0.2 on Centos 6.2 with OpenVZ and NFS.
Are there any tests that I could run to try and reproduce the problem ?

Comment by Andrew Luzin [ 08/Oct/11 ]

I made little investigation with this bug, and discussed it with OpenVZ developers. The reason is OpenVZ memory limitation mechanism. The problem is already fixed in OpenVZ in RHEL6 Kernel and some branches from RHEL6 kernel. In OpenVZ version earlier then RHEL6 this problem can't be fixed in generally.

Solutions:
For OpenVZ for RHEL6 or newer: Set physpages(http://wiki.openvz.org/VSwap) for container where mongoDB is running to limit amount of RAM, consumed by this container. Set privvmpages(http://wiki.openvz.org/UBC_secondary_parameters#privvmpages) for this container to unlimited or to huge number to allow MongoDB to map files into memory. I've tested this solution, it works.
For OpenVZ earlier than RHEL6 version there is no solution. You can set privvmpages parameter to unlimited and MongoDB will work, but it means that this container will have no limits to memory, and can capture all memory in your server. Don't do it if You not sure that You realy understand consequences.

Details:
OpenVZ with version older then RHEL6 had only one parameter for controlling the amount of memory allocated by container - privvmpages(http://wiki.openvz.org/UBC_secondary_parameters#privvmpages). This parameter limits number of virtual(not physical!) memory pages, that all processes in container can allocate. As you know, mapping file into memory causes allocating virtual memory pages. So, when MongoDB tries to map into memory files with total size more than privvmpages, other process can't allocate memory.
In the new OpenVZ(implemented in RHEL6, and available in new Proxmox and some other products) was added new memory management model - VSwap(http://wiki.openvz.org/VSwap). It allows to limit amount of physical(i.e. RAM) pages. This parameter named physpages. Since it, you can set limit for virtual pages to huge number or unlimited.

Links:
Discussion problem with OpenVZ developers: http://bugzilla.openvz.org/show_bug.cgi?id=2025
OpenVZ VSwap: http://wiki.openvz.org/VSwap
OpenVZ memory limitation parameters: http://wiki.openvz.org/UBC_secondary_parameters#privvmpages

Comment by kevinlau [ 26/May/11 ]

After studying many dbs, we decide to use mongoDB for our deployment but we can't continue our deployment without this fix. Any news or quick fix would be appreciated.

What are the pitfalls of this quick fix?
http://hachiari.com/blog/2011/03/31/getting-mongodb-to-work-on-openvz-without-out-of-memory-problem/

Comment by Juri Kuehn [ 02/Apr/11 ]

Could please someone post the link to the openvz bugtracker related to this issue? There should be one, but I could not find it.

Comment by Eliot Horowitz (Inactive) [ 11/Mar/11 ]

Durability was implemented with mmap, so has no impact on this issue.

Comment by Rich Thornett [ 11/Mar/11 ]

Looks like contact was made with the openvz team, so I wanted to check and see if there had been any progress on this. My site is hosted at Media Temple where all of their server offerings run on openvz (Virtuozzo). So presumably I (and the rest of their customers) are unable to run MongoDB without a fix.

To follow up on Richard Kreuter's comment above:

"This problem may go away if mongodb stops relying on VM, probably with server-980. Moving to 1.7"

The issue referenced, durability:

http://jira.mongodb.org/browse/SERVER-980

has been implemented and released in v1.7.5. However, I still see the memory allocation errors with 1.8rc-1. Having said that, would running the server with durability turned on (I have not to this point) change the memory management implementation such that this problem might go away? Or did implementing durability not require a move away from VM?

Thanks, folks. Great product, I hope I can use it!

Comment by Stuart Johnson [ 23/Feb/11 ]

LXC seems to be the new OpenVZ. It's part of the linux kernel, and major distros are switching to it. http://lxc.sourceforge.net

Does this problem exist on LXC?

Comment by Eliot Horowitz (Inactive) [ 22/Nov/10 ]

Working with openvz team.

Comment by ZedroS [ 06/Sep/10 ]

as seen there http://groups.google.com/group/mongodb-user/browse_thread/thread/4b756c95ded37297/ce08b362b65e79fc openvz allows trigger some count/iterators errors.

best

Comment by Jesse Proudman [ 06/Jul/10 ]

OpenVZ essentially throws the concept of virtual memory in the guest VM away, so your diagnosis seems accurate. Please let us know what else we can do to assist.

Comment by Richard Kreuter (Inactive) [ 06/Jul/10 ]

By inspection, OpenVZ seems to have a hard time with how mongod leans on virtual memory. Inserting a billion documents with integer ids eventually caused OpenVZ to run out of memory for new processes (i.e., fork() started to fail) and also for existing processes (i.e., existing shells started to die from malloc() errors). So we think OpenVZ's virtual memory isn't up to snuff. This problem may go away if mongodb stops relying on VM, probably with server-980. Moving to 1.7

http://jira.mongodb.org/browse/SERVER-980

Comment by Jesse Proudman [ 06/Jul/10 ]

Hi Richard,

Just sent you an email back with access to our cloud tools. Will continue to communicate via private email.

Thanks

  • Jesse
Comment by Richard Kreuter (Inactive) [ 06/Jul/10 ]

The thing that would be most helpful immediately would be access to an already running OpenVZ instance, especially one on which you've run into troubles with MongoDB. Please contact me at richard at 10gen dot com if you've some instances available. Thanks!

Comment by Jesse Proudman [ 06/Jul/10 ]

We're a hosting provider who primarily uses OpenVZ. We'd love to see resolution to this issue and would be happy to donate server space for testing. Please get in touch to let me know how we can be of assistance.

Thanks

  • Jesse
Generated at Thu Feb 08 02:56:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.