[SERVER-8006] mongod reports "Too many open files" after the first config server was killed and finally crashed Created: 21/Dec/12 Updated: 17/Jul/13 Resolved: 18/Jun/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.2.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | stronglee | Assignee: | Thomas Rueckstiess |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
os: Debian 6.0 kernel: Linux 2.6.38-2-amd64 |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
We tried to upgrade mongodb components from 2.2.1 to 2.2.2. (simply by kill & upgrade & start one by one component) The system's limit had already been set: I guess in this situation, mongod opens too many files/sockets and forget to close them. |
| Comments |
| Comment by stronglee [ 05/Mar/13 ] |
|
Hi Christopher Clarke, I think we didn't reboot the server at that time because we usually don't reboot servers only if they are down. |
| Comment by xofer [ 05/Mar/13 ] |
|
stronglee, I just want to make sure that you know the bug only affects limits on boot. If you start mongod any other way, the limits are set correctly. |
| Comment by stronglee [ 24/Dec/12 ] |
|
Hi Thomas Rueckstiess, Thanks for your reply. 1) We had set the limits in the right way just like the articles you recommend. 2) I use not only "ulimit -a" but also "cat /proc/<mongod pid>/limits" commands to see the limits, and both of them tell that the 3) I wrote a small python program to test the max open files. #!/usr/bin/python import os And run it with start-stop-daemon: start-stop-daemon --start --exec /home/strlee/test/f.py Traceback (most recent call last): In the normal state, our mongod opens about 3000 files/sockets at most. lsof | grep mongodb | wc -l So I think the mongod opened more than 65000 files/socketes when this bug occured. Thanks |
| Comment by Thomas Rueckstiess [ 24/Dec/12 ] |
|
Hi, Do you use start-stop-daemon to start the mongod process? There is a bug in Debian which I believe hasn't been fixed yet (even though it was first reported in 2005). Basically, it ignores the settings from the limits file for processes started as daemons. See also the following links that describe the problem and possible solutions: The last link recommends to add an explicit ulimits call to your init.d script before running mongod. Please could you confirm if you're affected by this Debian bug. Thanks, |
| Comment by stronglee [ 22/Dec/12 ] |
|
Hi Eliot Horowitz, please see the serverStatus.txt attachment. |
| Comment by Eliot Horowitz (Inactive) [ 22/Dec/12 ] |
|
Can you send db.serverStatus() then? |
| Comment by stronglee [ 22/Dec/12 ] |
|
hi Eliot Horowitz, I'm sorry that we didn't use mms for some reasons. But we deployed a monitoring system worked the same way as mms. |
| Comment by Eliot Horowitz (Inactive) [ 21/Dec/12 ] |
|
Is this node in mms? |