<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:03:48 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3705] mongos not closing connections, filling up disk with logs</title>
                <link>https://jira.mongodb.org/browse/SERVER-3705</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;One of our mongos started spewing out this message:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Mon Aug 29 00:40:36 [mongosMain] Listener: accept() returns -1 errno:24 Too many open files&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;it wrote 6.5 gigabytes of identical lines (same timestamp and all), which filled up the root partition of the server. All queries running through that mongos started failing at the same time, which led to the application failing. &lt;/p&gt;

&lt;p&gt;However, mongos is still running, and holding 300 TCP connections open (according to lsof).&lt;/p&gt;

&lt;p&gt;From what I can see it started about an hour earlier with mongos not being able to connect to the cluster, which goes on until it can connect to all the nodes except one, and then just minutes before it starts spewing out the messages about too many open files it manages to connect to the last one too. Then it writes the same message until the disk runs out.&lt;/p&gt;</description>
                <environment></environment>
        <key id="21606">SERVER-3705</key>
            <summary>mongos not closing connections, filling up disk with logs</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="iconara">Theo Hultberg</reporter>
                        <labels>
                    </labels>
                <created>Mon, 29 Aug 2011 05:11:38 +0000</created>
                <updated>Mon, 11 Jul 2016 18:35:40 +0000</updated>
                            <resolved>Fri, 2 Sep 2011 05:11:31 +0000</resolved>
                                    <version>1.8.2</version>
                                    <fixVersion>2.0.0-rc1</fixVersion>
                                                        <votes>1</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="51241" author="auto" created="Mon, 29 Aug 2011 17:21:07 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;login&apos;: u&apos;RedBeard0531&apos;, u&apos;name&apos;: u&apos;Mathias Stearn&apos;, u&apos;email&apos;: u&apos;mathias@10gen.com&apos;}
&lt;p&gt;Message: Wait a bit before trying to accept() when out of FDs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3707&quot; title=&quot;Don&amp;#39;t try to accept() if out of fds&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3707&quot;&gt;&lt;del&gt;SERVER-3707&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3705&quot; title=&quot;mongos not closing connections, filling up disk with logs&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3705&quot;&gt;&lt;del&gt;SERVER-3705&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3fc16f4e36aed091877448e9366b6b0949c686db&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3fc16f4e36aed091877448e9366b6b0949c686db&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="51192" author="eliot" created="Mon, 29 Aug 2011 15:25:59 +0000"  >&lt;p&gt;mongos has connection pools to shards and the config servers, so even when idle with no incoming connections, there will still be outgoing connections.&lt;/p&gt;

&lt;p&gt;If you&apos;re seeing entries like that it means queries are happening.&lt;/p&gt;</comment>
                            <comment id="51189" author="effata" created="Mon, 29 Aug 2011 15:19:49 +0000"  >&lt;p&gt;Since this in a new cluster we&apos;re breaking in we shut down the application completely. Mongos is completely idle but we still see 3-400 open connections from mongos to 27017 and config servers (3 shards and 3 config servers). It&apos;s been more than 15 minutes with no activity and the open connections have not changed.&lt;/p&gt;

&lt;p&gt;We also see many entries like this:&lt;br/&gt;
Mon Aug 29 14:59:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; ns: fragments_20110822.exposure_fragments ClusteredCursor::query ShardConnection had to change attempt: 0&lt;br/&gt;
Mon Aug 29 14:59:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; ns: fragments_20110822.pageview_fragments ClusteredCursor::query ShardConnection had to change attempt: 0&lt;/p&gt;</comment>
                            <comment id="51187" author="eliot" created="Mon, 29 Aug 2011 15:13:15 +0000"  >&lt;p&gt;50 is the number of incoming connections, the total number of tcp connections needed is more than that since we have to connect to every shard.&lt;br/&gt;
Please try increasing and see if it levels out.&lt;/p&gt;</comment>
                            <comment id="51177" author="iconara" created="Mon, 29 Aug 2011 14:57:28 +0000"  >&lt;p&gt;We run one application against the mongos, that application uses the Ruby driver and has a connection pool limit of 50. It seems unlikely that raising the limit to several thousand would do anything but postpone the inevitable.&lt;/p&gt;

&lt;p&gt;Then there&apos;s the issue of it writing gigabytes after gigabytes of logs, which makes it look even more like a bug in mongos.&lt;/p&gt;

&lt;p&gt;Since reporting this the error has happened again, and this time two mongos ran into the bug at more or less the same time.&lt;/p&gt;</comment>
                            <comment id="51171" author="eliot" created="Mon, 29 Aug 2011 14:51:26 +0000"  >&lt;p&gt;Is there any reason you think its not closing rather than just not having enough file descriptors?&lt;br/&gt;
Can you increase the fd limit to 16000?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="21623">SERVER-3706</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="21626">SERVER-3708</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="21625">SERVER-3707</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 29 Aug 2011 14:51:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 25 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 25 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>effata</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>iconara</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hroshb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrisrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23313</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0d3j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>