<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:04:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3776] Server unable to accept more than ~1000 connections due to new ulimit setting in RHEL 6</title>
                <link>https://jira.mongodb.org/browse/SERVER-3776</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;   I had an issue where my Mongo server would not accept more than 1024 connections after I upgraded from RHEL 5.6 to RHEL 6. (I also upgraded Mongo from 1.8.0 to 1.8.1 at the same time).  Apparently, RHEL 6 added a soft ulimit for processes of 1024, and Mongo threads count as processes. Mongo got kind of unstable when this happened - new connection attempts failed, but the connection count in serverStatus continued to increase. The server log showed connections accepted but did not indicate that they had failed. An strace of the server process showed the &quot;clone&quot; syscall failing with EAGAIN.&lt;/p&gt;

&lt;p&gt;   Since I was starting Mongo with an upstart script, not a shell, using the ulimit command would not work.&lt;/p&gt;

&lt;p&gt;   I fixed the problem by commenting out the &quot;nproc&quot; line in /etc/security/limits.d/90-nproc.conf . After that, the server worked OK.&lt;/p&gt;

&lt;p&gt;So, some questions:&lt;/p&gt;

&lt;p&gt;1)      Do you think that renaming the limits file with  &quot;mv /etc/security/limits.d/90-nproc.conf /etc/security/limits.d/90-nproc.conf.rpmsave&quot; is a good idea? That would be much easier to do in my RPM post-install script than editing the file as I did for the first system.&lt;/p&gt;

&lt;p&gt;2)      Has this been reported before? If so, is there some writeup you can point me to?&lt;/p&gt;

&lt;p&gt;3)      Can we request an improvement to Mongo so that it handles this situation a little better? It would have saved me an hour of troubleshooting if Mongo put a useful message in the server log when the &quot;clone&quot; syscall failed. Also it would be good to document that the ulimit for max processes must be greater than the largest number of threads that could be connected to the database, at least on this platform.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;br/&gt;
-john&lt;/p&gt;</description>
                <environment>RHEL 6, MongoDB 1.8.1</environment>
        <key id="21959">SERVER-3776</key>
            <summary>Server unable to accept more than ~1000 connections due to new ulimit setting in RHEL 6</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="richard.kreuter">Richard Kreuter</assignee>
                                    <reporter username="jfeibusc@ebay.com">John Feibusch</reporter>
                        <labels>
                    </labels>
                <created>Wed, 7 Sep 2011 17:10:37 +0000</created>
                <updated>Fri, 30 Mar 2012 14:37:08 +0000</updated>
                            <resolved>Fri, 6 Jan 2012 16:07:12 +0000</resolved>
                                                                    <component>Packaging</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="78146" author="richard@10gen.com" created="Fri, 6 Jan 2012 16:07:12 +0000"  >&lt;p&gt;mongodb can&apos;t do a whole lot if the operating system&apos;s settings are tuned too low.  Version 2.1 and greater now issue a warning for low resouce limits.&lt;/p&gt;</comment>
                            <comment id="70276" author="richard@10gen.com" created="Thu, 1 Dec 2011 23:02:23 +0000"  >&lt;p&gt;Logging warnings about bad rlimits might help.&lt;/p&gt;</comment>
                            <comment id="70275" author="richard@10gen.com" created="Thu, 1 Dec 2011 23:01:49 +0000"  >&lt;p&gt;This turns out not to be a packaging issue, but a dubiously low distro setting that absolutely needs changing on any host that runs MongoDB.&lt;/p&gt;</comment>
                            <comment id="53085" author="daqscott" created="Thu, 8 Sep 2011 21:41:19 +0000"  >&lt;p&gt;Another way around this is to use sudo. Add a rule to allow your mongo user id to run any command you like as itself with no password, then the ulimits will be set to your defaults in limits.conf&lt;/p&gt;</comment>
                            <comment id="53044" author="jfeibusc" created="Thu, 8 Sep 2011 19:26:27 +0000"  >&lt;p&gt;I have attached the upstart file and the limits.conf file. I can&apos;t provide the /proc/&amp;lt;pid&amp;gt;/limits file, since I have renamed the limits.conf file on all of our MongoDB servers.&lt;/p&gt;</comment>
                            <comment id="53042" author="jfeibusc" created="Thu, 8 Sep 2011 19:23:23 +0000"  >&lt;p&gt;This file is part of the base install of RHEL 6.&lt;/p&gt;</comment>
                            <comment id="53034" author="richard@10gen.com" created="Thu, 8 Sep 2011 19:01:39 +0000"  >&lt;p&gt;Yes, could you attach those 3 files anyway, so I can debug the settings?&lt;/p&gt;</comment>
                            <comment id="53024" author="jfeibusc" created="Thu, 8 Sep 2011 18:43:03 +0000"  >&lt;p&gt;Check out the doc for runuser in RHEL 6. It has changed. The man page in RHEL 6 says:&lt;br/&gt;
&quot;Change  the effective user id and group id to that of USER.  Only session PAM hooks are run, and there is no password prompt.&lt;/p&gt;

&lt;p&gt;In RHEL 5.6 it says:&lt;br/&gt;
&quot;Change the effective user id and group id to that of USER.  No PAM hooks are run, and there will be no password prompt.&quot;&lt;/p&gt;

&lt;p&gt;I don&apos;t know if the actual behavior has changed or if this is just a doc correction, but the current version seems to be behaving correctly according to the current documentation. Do you still want the files you mentioned?&lt;/p&gt;</comment>
                            <comment id="53022" author="richard@10gen.com" created="Thu, 8 Sep 2011 18:32:06 +0000"  >&lt;p&gt;That&apos;s weird, since the available documentation for runuser suggests that the point of it is not to use PAM.  Could you attach (1) the upstart config file you&apos;re using to start mongod, (2) your /etc/security/limits.d/90-nproc.conf, and (3) the contents of /proc/&amp;lt;pid&amp;gt;/limits, where &amp;lt;pid&amp;gt; is the pid of the mongod you&apos;ve started via your upstart config?  Thanks in advance.&lt;/p&gt;</comment>
                            <comment id="52859" author="jfeibusc" created="Wed, 7 Sep 2011 20:56:03 +0000"  >&lt;p&gt;Richard&apos;s idea sounded very good - I was slightly embarrassed for not having thought of it. But it doesn&apos;t seem to work. I start the Mongo server with a &quot;runuser&quot; command so that the server will run as the mongo user rather than as root. I think the upstart script in the 10gen package does the same thing. Unfortunately, &quot;runuser&quot; invokes the pam_limits module, which overrides any limits you set within the upstart script. I could run the &quot;ulimit&quot; command within the runuser command, but then it is running as a non-privileged user so the limit can&apos;t exceed the hard limit. So I would first have to determine what the hard limit is and then set the soft limit to that value. That&apos;s a bit more logic than I&apos;d like to have in an upstart script.&lt;/p&gt;</comment>
                            <comment id="52834" author="richard@10gen.com" created="Wed, 7 Sep 2011 18:36:42 +0000"  >&lt;p&gt;It should be possible (for sufficiently recent upstart versions, anyway) to increase the nproc limit in the upstart job itself.  If you&apos;ve a spare RHEL 6 system around to try this on (I haven&apos;t), could you add&lt;/p&gt;

&lt;p&gt;limit nproc 16000 16000&lt;/p&gt;

&lt;p&gt;to the upstart file and see if that helps?  This way, you don&apos;t need to touch the system-wide defaults in /etc/security.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="25847">SERVER-4407</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="12997" name="90-nproc.conf" size="152" author="jfeibusc" created="Thu, 8 Sep 2011 19:23:23 +0000"/>
                            <attachment id="12998" name="mongod.conf" size="582" author="jfeibusc" created="Thu, 8 Sep 2011 19:23:43 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 7 Sep 2011 17:36:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 6 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 6 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jfeibusc</customfieldvalue>
            <customfieldvalue>jfeibusc@ebay.com</customfieldvalue>
            <customfieldvalue>richard.kreuter</customfieldvalue>
            <customfieldvalue>daqscott</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrorlb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriqav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22914</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0bxr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>