<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 02:57:36 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-1633] too many open files, lsof &quot;can&apos;t identify protocol&quot;</title>
                <link>https://jira.mongodb.org/browse/SERVER-1633</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;on all of our 80 appserver, 1 mongos-Daemon is installed. Very frequently, the mongos-Daemon stops working because of &quot;too many open files&quot;, which is also spamming the mongos-logfiles. We increased the openfiles limit to 30000 and this fixed the problem for a short time. But soon the problem was there again, and I discovered the real source. Somehow the mongos-Daemons seems to create some kind of idleing zombie-connections. When I do a &lt;/p&gt;

&lt;p&gt;lsof | grep &apos;mongos&apos;&lt;/p&gt;

&lt;p&gt;This is the output:&lt;br/&gt;
......thousands of lines .....&lt;br/&gt;
mongos    22843      root 8523u     sock                0,5                     2863975141 can&apos;t identify protocol&lt;br/&gt;
mongos    22843      root 8524u     sock                0,5                     2863975143 can&apos;t identify protocol&lt;br/&gt;
mongos    22843      root 8525u     sock                0,5                     2863975162 can&apos;t identify protocol&lt;br/&gt;
mongos    22843      root 8526u     sock                0,5                     2863975219 can&apos;t identify protocol&lt;br/&gt;
......thousands of lines ......&lt;/p&gt;


&lt;p&gt;These zombie-connections fill up all the available filedescriptors.&lt;/p&gt;

&lt;p&gt;For me it looks like a bug in mongos , or the mongodb-java connector (or both).&lt;/p&gt;</description>
                <environment>we use mongodb with sharding and the java-connector&lt;br/&gt;
80 appservers (tomcat), on every appserver 1 mongos-daemon, connecting to 2 mongod-server</environment>
        <key id="12822">SERVER-1633</key>
            <summary>too many open files, lsof &quot;can&apos;t identify protocol&quot;</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="alerner">Alberto Lerner</assignee>
                                    <reporter username="jonasgk">Jonas</reporter>
                        <labels>
                    </labels>
                <created>Wed, 18 Aug 2010 11:20:39 +0000</created>
                <updated>Tue, 29 May 2012 14:53:32 +0000</updated>
                            <resolved>Mon, 6 Sep 2010 14:22:09 +0000</resolved>
                                    <version>1.6.0</version>
                                                    <component>Sharding</component>
                    <component>Stability</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="17664" author="alerner" created="Wed, 1 Sep 2010 12:55:33 +0000"  >&lt;p&gt;Solving this problem &amp;#8211; which smells like a single bug &amp;#8211; is of the highest priority to us. Your help has been instrumental to isolate it. We appreciate you sending this log and apologize that, besides our best effort, we haven&apos;t gotten to the bottom of this.&lt;/p&gt;

&lt;p&gt;The log you sent has  one more trail I&apos;d like to drill down, so that&apos;s helpful. But we added instrumentation code to the 1.7 branch.&lt;/p&gt;

&lt;p&gt;I&apos;ll keep you posted.&lt;/p&gt;</comment>
                            <comment id="17658" author="jonasgk" created="Wed, 1 Sep 2010 07:58:18 +0000"  >&lt;p&gt;Hi, I&apos;m sorry to tell you that for now we suspended our tests with mongodb, as for our needs the sharding-functionality was too unstable, we couldn&apos;t find the reason and we can&apos;t invest any more time. With todays deployment we deactivated mongodb. However yesterday I deployed the nightly-version and I attached the gzipped-logfile of this one day. Perhaps it can be of use.&lt;/p&gt;</comment>
                            <comment id="17524" author="eliot" created="Sat, 28 Aug 2010 17:40:08 +0000"  >&lt;p&gt;The nightly is up - so you can give it another shot?&lt;/p&gt;

&lt;p&gt;Thanks for all your patience - really want to figure this out.&lt;/p&gt;</comment>
                            <comment id="17467" author="auto" created="Thu, 26 Aug 2010 19:08:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;login&apos;: &apos;alerner&apos;, &apos;name&apos;: &apos;Alberto Lerner&apos;, &apos;email&apos;: &apos;alerner@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-1633&quot; title=&quot;too many open files, lsof &amp;quot;can&amp;#39;t identify protocol&amp;quot;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-1633&quot;&gt;&lt;del&gt;SERVER-1633&lt;/del&gt;&lt;/a&gt; turn logging up to uncover bottleneck&lt;br/&gt;
&lt;a href=&quot;http://github.com/mongodb/mongo/commit/9aac39189e67a0abcb84a2c311c981f435c3ffbd&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://github.com/mongodb/mongo/commit/9aac39189e67a0abcb84a2c311c981f435c3ffbd&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="17459" author="alerner" created="Thu, 26 Aug 2010 16:36:43 +0000"  >&lt;p&gt;We got a good lead using the last log. But we still need to dig deeper to find the source of the problem.&lt;/p&gt;

&lt;p&gt;Two questions: &lt;/p&gt;

&lt;p&gt;Might anything out of the ordinary happened Aug 25 at around 8:15 pm on? We got an exponential response time from a particular internal command (setShardVersion) that starts then. &lt;/p&gt;

&lt;p&gt;What is the platform you&apos;re running on? I&apos;ll put some changes that will help us peel more layers using the log. If you would be okay with it, they&apos;d get in todays nightly pipeline.&lt;/p&gt;</comment>
                            <comment id="17447" author="jonasgk" created="Thu, 26 Aug 2010 06:58:33 +0000"  >&lt;p&gt;the network between the database and the mongos is fine. I attached the anonymized log of the failing shard server. It runs for 1 or 2 days, and then it starts failing (too many connections ...).&lt;/p&gt;</comment>
                            <comment id="17195" author="alerner" created="Mon, 23 Aug 2010 18:42:46 +0000"  >&lt;p&gt;Jonas, the log indicates a log of connectivity issues. The high number of &quot;create&quot; in connPoolStats reflect it; the mongos reconnecting after getting broken pipes. We were wondering why those are happening. Is the networking bad between mongos and XX.XX.XX.87:3307? Or is that shard crashing?&lt;/p&gt;

&lt;p&gt;We still can&apos;t reproduce this here.&lt;/p&gt;</comment>
                            <comment id="17168" author="jonasgk" created="Mon, 23 Aug 2010 12:19:34 +0000"  >&lt;p&gt;On sv287 there are again these huge connection leaks. I attached the mongos.log . &lt;/p&gt;

&lt;p&gt;MongoDB shell version: 1.6.0&lt;br/&gt;
connecting to: 127.0.0.1:3309/test&lt;br/&gt;
&amp;gt; db.runCommand( &quot;connPoolStats&quot; )&lt;br/&gt;
{&lt;br/&gt;
	&quot;hosts&quot; : {&lt;br/&gt;
		&quot;XX.XX.XX.86:3307&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 1,
			&quot;created&quot; : 357
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;XX.XX.XX.87:3307&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 0,
			&quot;created&quot; : 19516
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;XX.XX.XX.87:3308&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 1,
			&quot;created&quot; : 1593
		}
&lt;p&gt;	},&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="17051" author="eliot" created="Thu, 19 Aug 2010 16:14:10 +0000"  >&lt;p&gt;Ok - it looks like we&apos;re leaking connections to BBBBB&lt;br/&gt;
Can you attach the a full mongos log?&lt;/p&gt;</comment>
                            <comment id="17047" author="jonasgk" created="Thu, 19 Aug 2010 15:49:44 +0000"  >&lt;p&gt;I run the command on one appserver. The very high number of created connections to shard2 (BBBB) is almost the same like the idling &quot;zombie&quot; connections. (its fairly low (about 2000) now because of a recent restart, i will check again in some hours and post the result). I also posted our mongodb config for shard2.&lt;/p&gt;

&lt;p&gt;on port 3307 the shards are listening&lt;br/&gt;
AAAA = ip of first mongodb shard&lt;br/&gt;
BBBB = ip of second mongodb shard&lt;br/&gt;
CCCC = the mongodb config server&lt;/p&gt;

&lt;p&gt;&amp;gt; db.runCommand( &quot;connPoolStats&quot; )&lt;br/&gt;
{&lt;br/&gt;
	&quot;hosts&quot; : {&lt;br/&gt;
		&quot;AAAAA:3307&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 1,
			&quot;created&quot; : 36
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;BBBBB:3307&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 0,
			&quot;created&quot; : 1915
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;CCCCC:3308&quot; : &lt;/p&gt;
{
			&quot;available&quot; : 0,
			&quot;created&quot; : 75
		}
&lt;p&gt;	},&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="17032" author="eliot" created="Thu, 19 Aug 2010 13:20:18 +0000"  >&lt;p&gt;Can you connect to the mongos and run&lt;/p&gt;

&lt;p&gt;db.runCommand( &quot;connPoolStats&quot; )&lt;/p&gt;</comment>
                            <comment id="17030" author="jonasgk" created="Thu, 19 Aug 2010 12:48:27 +0000"  >&lt;p&gt;This are the commands to generate the files (i attached the files to this ticket)&lt;/p&gt;

&lt;p&gt;XXX.XXX.XXX.XXX is the IP of the appserver&lt;br/&gt;
YYY.YYY.YYY.YYY is the IP of the mongodb-server&lt;br/&gt;
netstat -tn | grep 3307 &amp;gt; netstat_grep3307.txt&lt;/p&gt;

&lt;p&gt;The mongos is listening on port 3309, so this are the connections from appserver to mongos&lt;br/&gt;
netstat -tn | grep 3309 &amp;gt; netstat_grep3309.txt&lt;/p&gt;

&lt;p&gt;a linecount on the files:&lt;br/&gt;
  1183 netstat_grep3307.txt&lt;br/&gt;
   370 netstat_grep3309.txt&lt;/p&gt;

&lt;p&gt;lsof | grep mongos | grep -c &quot;can&apos;t identify protocol&quot;&lt;br/&gt;
11561&lt;/p&gt;

&lt;p&gt;The developers said, that the java-mongodb-driver is set to connection pooling and should only open like 75 connections maximum.&lt;/p&gt;</comment>
                            <comment id="17013" author="eliot" created="Wed, 18 Aug 2010 17:40:57 +0000"  >&lt;p&gt;I&apos;m looking for the socket states and what they are connection too.&lt;/p&gt;

&lt;p&gt;I..e mongos -&amp;gt; mongod, or app -&amp;gt; mongos&lt;/p&gt;</comment>
                            <comment id="17012" author="jonasgk" created="Wed, 18 Aug 2010 17:38:44 +0000"  >&lt;p&gt;this is our mongos command:&lt;br/&gt;
/opt/mongo/bin/mongos --port 3309 --quiet --configdb &amp;lt;IPOFCONFIGDB&amp;gt;:3308&lt;/p&gt;

&lt;p&gt;What is your theory, or what exactly are you looking for? Perhaps with this information i can gather more useful information.&lt;/p&gt;</comment>
                            <comment id="17011" author="eliot" created="Wed, 18 Aug 2010 17:32:31 +0000"  >&lt;p&gt;Are you using normal mongo ports?&lt;/p&gt;</comment>
                            <comment id="17010" author="jonasgk" created="Wed, 18 Aug 2010 17:28:58 +0000"  >&lt;p&gt;Here is the netstat output:&lt;/p&gt;

&lt;p&gt;&quot;netstat -an | grep 270&quot;&lt;br/&gt;
tcp        1      0 127.0.0.1:3309              127.0.0.1:42709             CLOSE_WAIT  &lt;br/&gt;
tcp        1      0 127.0.0.1:3309              127.0.0.1:52706             CLOSE_WAIT &lt;/p&gt;

&lt;p&gt;and here the number currently opened &quot;zombie connections&quot;&lt;br/&gt;
lsof | grep &quot;mongos&quot; | grep -c &quot;can&apos;t identify protocol&quot;&lt;br/&gt;
15152&lt;/p&gt;

&lt;p&gt;Some additional information about other TCP-Connection states&lt;br/&gt;
netstat -nt | awk &apos;&lt;/p&gt;
{print $6}
&lt;p&gt;&apos; | sort | uniq -c | sort -n -k 1 -r&lt;br/&gt;
   1832 ESTABLISHED&lt;br/&gt;
    633 FIN_WAIT2&lt;br/&gt;
    504 TIME_WAIT&lt;br/&gt;
    466 CLOSE_WAIT&lt;br/&gt;
     11 FIN_WAIT1&lt;br/&gt;
      2 SYN_SENT&lt;br/&gt;
      1 Foreign&lt;br/&gt;
      1 &lt;/p&gt;</comment>
                            <comment id="17006" author="eliot" created="Wed, 18 Aug 2010 16:44:50 +0000"  >&lt;p&gt;Can you do &quot;netstat -an | grep 270&quot; on the appservers and attach that&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="10417" name="mongod-2010-09-01.log.gz" size="2346734" author="jonasgk" created="Wed, 1 Sep 2010 07:58:18 +0000"/>
                            <attachment id="10385" name="mongod.conf" size="1870" author="jonasgk" created="Thu, 19 Aug 2010 15:49:44 +0000"/>
                            <attachment id="10400" name="mongod.log.2010-08-25.gz" size="2153786" author="jonasgk" created="Thu, 26 Aug 2010 06:58:33 +0000"/>
                            <attachment id="10387" name="mongos_debug.log.gz" size="17661" author="jonasgk" created="Mon, 23 Aug 2010 12:19:34 +0000"/>
                            <attachment id="10382" name="netstat_grep3307.txt" size="108701" author="jonasgk" created="Thu, 19 Aug 2010 12:48:27 +0000"/>
                            <attachment id="10383" name="netstat_grep3309.txt" size="32930" author="jonasgk" created="Thu, 19 Aug 2010 12:48:27 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>17.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 18 Aug 2010 16:44:50 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        13 years, 25 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            13 years, 25 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alerner</customfieldvalue>
            <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>jonasgk</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrijpr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriipr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21684</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hria2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>