I had an issue where my Mongo server would not accept more than 1024 connections after I upgraded from RHEL 5.6 to RHEL 6. (I also upgraded Mongo from 1.8.0 to 1.8.1 at the same time). Apparently, RHEL 6 added a soft ulimit for processes of 1024, and Mongo threads count as processes. Mongo got kind of unstable when this happened - new connection attempts failed, but the connection count in serverStatus continued to increase. The server log showed connections accepted but did not indicate that they had failed. An strace of the server process showed the "clone" syscall failing with EAGAIN.

Since I was starting Mongo with an upstart script, not a shell, using the ulimit command would not work.

I fixed the problem by commenting out the "nproc" line in /etc/security/limits.d/90-nproc.conf . After that, the server worked OK.

So, some questions:

1) Do you think that renaming the limits file with "mv /etc/security/limits.d/90-nproc.conf /etc/security/limits.d/90-nproc.conf.rpmsave" is a good idea? That would be much easier to do in my RPM post-install script than editing the file as I did for the first system.

2) Has this been reported before? If so, is there some writeup you can point me to?

3) Can we request an improvement to Mongo so that it handles this situation a little better? It would have saved me an hour of troubleshooting if Mongo put a useful message in the server log when the "clone" syscall failed. Also it would be good to document that the ulimit for max processes must be greater than the largest number of threads that could be connected to the database, at least on this platform.

Thanks!
-john