<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:00:30 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-2599] potential connection leak in sharded environment</title>
                <link>https://jira.mongodb.org/browse/SERVER-2599</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I have a system setup that is very similar to the architecture described in &lt;a href=&quot;http://www.snailinaturtleneck.com/blog/2010/08/30/return-of-the-mongo-mailbag/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.snailinaturtleneck.com/blog/2010/08/30/return-of-the-mongo-mailbag/&lt;/a&gt;&lt;br/&gt;
? 4 servers, 2 shards.&lt;br/&gt;
A disturbing fact is that the number of open connections to the two masters seems to be constantly increasing over time (see the attached ganglia graph).&lt;/p&gt;

&lt;p&gt;So here are a few weird observations that I made:&lt;br/&gt;
1) One mongos has less than 20 client connections but over 30 connections to &lt;b&gt;each&lt;/b&gt; mongod&lt;br/&gt;
2) The number of open connections reported by netstat is much less than the number of connections reported in db.serverStatus(): 50 vs. 101&lt;br/&gt;
3) db.currentOps() shows a few active &quot;writebacklisten&quot; operations that are running for 3+ days (might be related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-2434&quot; title=&quot;Filter &amp;quot;db.currentOp()&amp;quot; to remove writebacklisten ops and more&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-2434&quot;&gt;&lt;del&gt;SERVER-2434&lt;/del&gt;&lt;/a&gt;)&lt;br/&gt;
4) the mongos log contains a couple of client connection timeout errors like: &quot;MessagingPort recv() errno:110 Connection timed out...&quot;&lt;/p&gt;

&lt;p&gt;The clients use the Java driver in version 2.4 and the client instance is kept open and reused for each query.&lt;/p&gt;</description>
                <environment># lsb_release -a&lt;br/&gt;
LSB Version:	n/a&lt;br/&gt;
Distributor ID:	SUSE LINUX&lt;br/&gt;
Description:	openSUSE 11.2 (x86_64)&lt;br/&gt;
Release:	11.2&lt;br/&gt;
Codename:	n/a</environment>
        <key id="14829">SERVER-2599</key>
            <summary>potential connection leak in sharded environment</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="bwaldvogel">Benedikt Waldvogel</reporter>
                        <labels>
                    </labels>
                <created>Mon, 21 Feb 2011 14:21:16 +0000</created>
                <updated>Thu, 17 Mar 2011 19:13:13 +0000</updated>
                            <resolved>Mon, 21 Feb 2011 14:25:11 +0000</resolved>
                                    <version>1.6.5</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>0</watches>
                                                                                                                <comments>
                            <comment id="24824" author="eliot" created="Mon, 28 Feb 2011 15:32:21 +0000"  >&lt;p&gt;The server sets keep alive, so its not strictly required on the java side, added a case though &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-287&quot; title=&quot;set TCP_KEEPALIVE on the java driver&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-287&quot;&gt;&lt;del&gt;JAVA-287&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;I believe serverStatus and netstat disagree because the mongod process isn&apos;t notified in the same way the tcp/ip stack is.&lt;/p&gt;</comment>
                            <comment id="24816" author="bwaldvogel" created="Mon, 28 Feb 2011 10:29:16 +0000"  >&lt;p&gt;That&apos;s true and I agree that it&apos;s a bad behavior of the firewall.&lt;br/&gt;
Nevertheless, it would definitely help if the Java driver would set keep alive on the socket (which it currently does not!).&lt;br/&gt;
I&apos;m also wondering why db.serverStatus() in the meantime reports 162 connections while netstat shows only 88. Is it a bug?&lt;/p&gt;</comment>
                            <comment id="24719" author="eliot" created="Fri, 25 Feb 2011 23:34:53 +0000"  >&lt;p&gt;Keep alive is set.&lt;/p&gt;

&lt;p&gt;Default kernel keep alive settings kick in at 2 hours.&lt;/p&gt;

&lt;p&gt;Routers aren&apos;t supposed to kill idle connections for 2 hours in general.&lt;/p&gt;</comment>
                            <comment id="24680" author="bwaldvogel" created="Fri, 25 Feb 2011 17:14:30 +0000"  >&lt;p&gt;Turns out that the frequent connection timeouts happen because the firewall silently drops connections that are idle for at least one hour.&lt;br/&gt;
I&apos;m surprised that the Java driver and mongos keep idle connections open for such a long time and don&apos;t even set TCP keep-alive.&lt;/p&gt;</comment>
                            <comment id="24521" author="bwaldvogel" created="Wed, 23 Feb 2011 15:44:44 +0000"  >&lt;p&gt;I&apos;ve attached the logfile.&lt;/p&gt;</comment>
                            <comment id="24435" author="eliot" created="Tue, 22 Feb 2011 16:27:45 +0000"  >&lt;p&gt;Can you attach the mongos log?&lt;/p&gt;</comment>
                            <comment id="24425" author="bwaldvogel" created="Tue, 22 Feb 2011 13:09:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://gist.github.com/838638#file_mongod_netstat.txt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/838638#file_mongod_netstat.txt&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://gist.github.com/838638#file_mongod_server_status.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/838638#file_mongod_server_status.js&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://gist.github.com/838638#file_mongos_conn_pool_stats.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/838638#file_mongos_conn_pool_stats.js&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://gist.github.com/838638#file_mongod_conn_pool_stats.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/838638#file_mongod_conn_pool_stats.js&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="24390" author="eliot" created="Mon, 21 Feb 2011 22:04:01 +0000"  >&lt;p&gt;Can you show serverStatus(), connPoolStats and netstat from the same point in time?&lt;/p&gt;</comment>
                            <comment id="24368" author="bwaldvogel" created="Mon, 21 Feb 2011 15:07:29 +0000"  >&lt;p&gt;how do you explain that db.serverStatus() reports over 100 connections while netstat shows only 50?&lt;/p&gt;</comment>
                            <comment id="24364" author="eliot" created="Mon, 21 Feb 2011 14:25:11 +0000"  >&lt;p&gt;mongos keeps connections open to all shards all the time.&lt;/p&gt;

&lt;p&gt;You can run&lt;br/&gt;
db.runCommand( &quot;connPoolStats&quot; )&lt;/p&gt;

&lt;p&gt;To see&lt;/p&gt;

&lt;p&gt;The writeback connections are also supposed to be there, and live forever. &lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="10856" name="mongodb_connection_leak.png" size="27170" author="bwaldvogel" created="Mon, 21 Feb 2011 14:21:16 +0000"/>
                            <attachment id="10865" name="mongos.log" size="111976" author="bwaldvogel" created="Tue, 22 Feb 2011 16:39:32 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 21 Feb 2011 14:25:11 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 51 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 51 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bwaldvogel</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrp5l3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hridin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>20842</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0g9b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>