<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:16:17 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-8006] mongod reports &quot;Too many open files&quot; after the first config server was killed and finally crashed</title>
                <link>https://jira.mongodb.org/browse/SERVER-8006</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We tried to upgrade mongodb components from 2.2.1 to 2.2.2. (simply by kill &amp;amp; upgrade &amp;amp; start one by one component)&lt;br/&gt;
Several minutes after we killed the first config server, one of mongod began to report &quot;Too many open files&quot;, and it crashed several minutes later.&lt;/p&gt;

&lt;p&gt;The system&apos;s limit had already been set:&lt;br/&gt;
Limit                     Soft Limit           Hard Limit           Units     &lt;br/&gt;
Max open files            65535                65535                files &lt;/p&gt;

&lt;p&gt;I guess in this situation, mongod opens too many files/sockets and forget to close them.&lt;br/&gt;
Please look into the attachement, it contains full log from the time we kill the config server to the end.&lt;/p&gt;</description>
                <environment>os: Debian 6.0 kernel: Linux 2.6.38-2-amd64 &lt;br/&gt;
cluster: a sharded cluster with 4 replica sets. Each rs contains a primary &amp;amp; a second &amp;amp; a arbiter (mongod v2.2.1) &lt;br/&gt;
</environment>
        <key id="60115">SERVER-8006</key>
            <summary>mongod reports &quot;Too many open files&quot; after the first config server was killed and finally crashed</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="thomas.rueckstiess@mongodb.com">Thomas Rueckstiess</assignee>
                                    <reporter username="strlee">stronglee</reporter>
                        <labels>
                    </labels>
                <created>Fri, 21 Dec 2012 08:19:50 +0000</created>
                <updated>Wed, 17 Jul 2013 16:38:25 +0000</updated>
                            <resolved>Tue, 18 Jun 2013 01:19:43 +0000</resolved>
                                    <version>2.2.1</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="281490" author="strlee" created="Tue, 5 Mar 2013 04:03:01 +0000"  >&lt;p&gt;Hi Christopher Clarke,&lt;/p&gt;

&lt;p&gt;   I think we didn&apos;t reboot the server at that time because we usually don&apos;t reboot servers only if they are down.&lt;br/&gt;
But it&apos;s really a long time from the bug happened so I am not 100 percent sure of this.&lt;br/&gt;
We didn&apos;t meet this bug again. Shall I suggest closing this issue for the reason &quot;Can not be replicated&quot; ?&lt;/p&gt;</comment>
                            <comment id="281478" author="xofer" created="Tue, 5 Mar 2013 03:40:16 +0000"  >&lt;p&gt;stronglee, I just want to make sure that you know the bug only affects limits on boot. If you start mongod any other way, the limits are set correctly. &lt;/p&gt;</comment>
                            <comment id="223395" author="strlee" created="Mon, 24 Dec 2012 15:38:15 +0000"  >&lt;p&gt;Hi Thomas Rueckstiess,&lt;/p&gt;

&lt;p&gt;Thanks for your reply.&lt;br/&gt;
We do use start-stop-daemon to start mongod.&lt;br/&gt;
However, I am quite sure that our system isn&apos;t affected by the Debian bug you described.&lt;/p&gt;

&lt;p&gt;1) We had set the limits in the right way just like the articles you recommend.&lt;/p&gt;

&lt;p&gt;2) I use not only &quot;ulimit -a&quot; but also &quot;cat /proc/&amp;lt;mongod pid&amp;gt;/limits&quot; commands to see the limits, and both of them tell that the &lt;br/&gt;
&quot;open files&quot; is 65535.&lt;/p&gt;

&lt;p&gt;3) I wrote a small python program to test the max open files.&lt;/p&gt;

&lt;p&gt;#!/usr/bin/python&lt;/p&gt;

&lt;p&gt;import os&lt;br/&gt;
os.chdir(&apos;/home/strlee/test/files&apos;)&lt;br/&gt;
l = []&lt;br/&gt;
for i in xrange(70000):&lt;br/&gt;
        l.append(open(str&lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/information.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;, &apos;w&apos;))&lt;/p&gt;

&lt;p&gt;And run it with start-stop-daemon: start-stop-daemon --start --exec /home/strlee/test/f.py&lt;br/&gt;
The result is:&lt;/p&gt;

&lt;p&gt;Traceback (most recent call last):&lt;br/&gt;
  File &quot;/home/strlee/test/f.py&quot;, line 8, in &amp;lt;module&amp;gt;&lt;br/&gt;
IOError: &lt;span class=&quot;error&quot;&gt;&amp;#91;Errno 24&amp;#93;&lt;/span&gt; Too many open files: &apos;65532&apos;&lt;/p&gt;


&lt;p&gt;In the normal state, our mongod opens about 3000 files/sockets at most.&lt;/p&gt;

&lt;p&gt;lsof | grep mongodb | wc -l&lt;br/&gt;
2728&lt;/p&gt;

&lt;p&gt;So I think the mongod opened more than 65000 files/socketes when this bug occured.&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;</comment>
                            <comment id="223162" author="thomasr" created="Mon, 24 Dec 2012 06:02:33 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;Do you use &lt;tt&gt;start-stop-daemon&lt;/tt&gt; to start the mongod process? &lt;/p&gt;

&lt;p&gt;There is a &lt;a href=&quot;http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bug&lt;/a&gt; in Debian which I believe hasn&apos;t been fixed yet (even though it was first reported in 2005). Basically, it ignores the settings from the &lt;tt&gt;limits&lt;/tt&gt; file for processes started as daemons. &lt;/p&gt;

&lt;p&gt;See also the following links that describe the problem and possible solutions:&lt;br/&gt;
&lt;a href=&quot;http://www.jayway.com/2012/02/11/how-to-really-fix-the-too-many-open-files-problem-for-tomcat-in-ubuntu/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.jayway.com/2012/02/11/how-to-really-fix-the-too-many-open-files-problem-for-tomcat-in-ubuntu/&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://ubuntuforums.org/showthread.php?t=1583041&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://ubuntuforums.org/showthread.php?t=1583041&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://superuser.com/questions/454465/make-ulimits-work-with-start-stop-daemon&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://superuser.com/questions/454465/make-ulimits-work-with-start-stop-daemon&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The last link recommends to add an explicit ulimits call to your &lt;tt&gt;init.d&lt;/tt&gt; script before running mongod.&lt;/p&gt;

&lt;p&gt;Please could you confirm if you&apos;re affected by this Debian bug.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="222030" author="strlee" created="Sat, 22 Dec 2012 07:24:17 +0000"  >&lt;p&gt;Hi Eliot Horowitz, please see the serverStatus.txt attachment.&lt;/p&gt;</comment>
                            <comment id="222006" author="eliot" created="Sat, 22 Dec 2012 06:22:23 +0000"  >&lt;p&gt;Can you send db.serverStatus() then?&lt;/p&gt;</comment>
                            <comment id="221942" author="strlee" created="Sat, 22 Dec 2012 04:08:04 +0000"  >&lt;p&gt;hi Eliot Horowitz, I&apos;m sorry that we didn&apos;t use mms for some reasons. But we deployed a monitoring system worked the same way as mms.&lt;/p&gt;</comment>
                            <comment id="221354" author="eliot" created="Fri, 21 Dec 2012 14:46:22 +0000"  >&lt;p&gt;Is this node in mms?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="22773" name="log.txt" size="1047965" author="strlee" created="Fri, 21 Dec 2012 08:19:50 +0000"/>
                            <attachment id="22796" name="serverStatus.txt" size="72780" author="strlee" created="Sat, 22 Dec 2012 07:18:38 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 21 Dec 2012 14:46:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 50 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 50 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>strlee</customfieldvalue>
            <customfieldvalue>thomas.rueckstiess@mongodb.com</customfieldvalue>
            <customfieldvalue>xofer</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrndqf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrl7jb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>37562</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht024n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>