<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26740] Total connections is not stable and spiky when user authentication is enabled</title>
                <link>https://jira.mongodb.org/browse/SERVER-26740</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;MongoDB cluster is consist of 28 shards and 15 mongos.&lt;br/&gt;
There&apos;s about 4K/sec queries per shard and small DMLs and there&apos;s about 600~1000 connections(from mongos) per shard.&lt;/p&gt;

&lt;p&gt;MongoDB connections graph is spiky only when user authentication is enabled Not with disabled authentication. And java client driver return errors(like below) or slow-responses when user authentication is enabled. (I attached connections and query/sec graphs).&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;ERROR&amp;#93;&lt;/span&gt; &lt;span class=&quot;error&quot;&gt;&amp;#91;c.k.s.m.g.r.RequestExecuteCallable&amp;#93;&lt;/span&gt; execute (71): Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 400 has been exceeded.&lt;br/&gt;
com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 400 has been exceeded.&lt;/p&gt;


&lt;p&gt;Not exactly 5 minutes, but connection spike(and client queue error) is happened about 5 minutes intervals.&lt;br/&gt;
I am not sure &quot;ConnectionPool::kDefaultHostTimeout&quot; cause this issue yet, But I think 5 minutes is too short to shutdown connection pool. it seems so aggressive. Is there any reason to choose 5 minutes to shutdown pool ?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/executor/connection_pool.cpp#L177&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/master/src/mongo/executor/connection_pool.cpp#L177&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="326287">SERVER-26740</key>
            <summary>Total connections is not stable and spiky when user authentication is enabled</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="spencer.jackson@mongodb.com">Spencer Jackson</assignee>
                                    <reporter username="sunguck.lee@gmail.com">&#50500;&#45208; &#54616;&#47532;</reporter>
                        <labels>
                    </labels>
                <created>Mon, 24 Oct 2016 12:20:40 +0000</created>
                <updated>Wed, 8 Feb 2023 15:40:56 +0000</updated>
                            <resolved>Fri, 9 Jun 2017 18:12:59 +0000</resolved>
                                    <version>3.2.10</version>
                                                    <component>Networking</component>
                    <component>Security</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1593258" author="thomas.schubert" created="Fri, 9 Jun 2017 18:12:43 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We&apos;ve seen significant improvement for users running with &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26952&quot; title=&quot;Cache SCRAM-SHA-1 ClientKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26952&quot;&gt;&lt;del&gt;SERVER-26952&lt;/del&gt;&lt;/a&gt;. This fix is included in MongoDB 3.4.4 and later, and we&apos;re evaluating the backport to MongoDB 3.2. Please watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26952&quot; title=&quot;Cache SCRAM-SHA-1 ClientKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26952&quot;&gt;&lt;del&gt;SERVER-26952&lt;/del&gt;&lt;/a&gt; for updates.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1430230" author="jason.carey" created="Wed, 9 Nov 2016 15:54:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I had a conversation with an engineer on our security team and he had an idea he&apos;s written down in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26952&quot; title=&quot;Cache SCRAM-SHA-1 ClientKey&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26952&quot;&gt;&lt;del&gt;SERVER-26952&lt;/del&gt;&lt;/a&gt; (Caching scram credentials for connection pooling).  That change is very likely to help mitigate the kind of situation you&apos;ve seen in this ticket.&lt;/p&gt;

&lt;p&gt;While it won&apos;t make it for 3.4.0, it&apos;s an important change that I&apos;ll definitely be considering for a backport.&lt;/p&gt;</comment>
                            <comment id="1429355" author="jason.carey" created="Tue, 8 Nov 2016 16:46:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;My apologies, I hadn&apos;t updated the description of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25027&quot; title=&quot;Configurable connection pools size for mongos&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25027&quot;&gt;&lt;del&gt;SERVER-25027&lt;/del&gt;&lt;/a&gt; yet.  That ticket was re-purposed for providing the actual options available in mongos, and it will cover minConnections and hostTimeout.&lt;/p&gt;</comment>
                            <comment id="1428790" author="sunguck.lee@gmail.com" created="Tue, 8 Nov 2016 01:22:06 +0000"  >&lt;p&gt;Thanks Jason.&lt;/p&gt;

&lt;p&gt;So you think connection spike is expected when client authentication is enabled.&lt;/p&gt;

&lt;p&gt;And,&lt;br/&gt;
I think we need to knob to control minConnection and hostTimeout of connection pool (SpecificPool) in mongos rather than &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25027&quot; title=&quot;Configurable connection pools size for mongos&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25027&quot;&gt;&lt;del&gt;SERVER-25027&lt;/del&gt;&lt;/a&gt;.&lt;br/&gt;
Like I&apos;ve submitted in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26720&quot; title=&quot;Need to prepare initial connection between mongos and mongod&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26720&quot;&gt;SERVER-26720&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Matt.&lt;/p&gt;
</comment>
                            <comment id="1428573" author="jason.carey" created="Mon, 7 Nov 2016 21:57:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Regarding the HostTimeoutMS, that represents the number of milliseconds to keep a pool alive, and min connections (default 1) open &lt;b&gt;without activity&lt;/b&gt;.  So if you&apos;re seeing spikes in this way, it would be for hosts you&apos;re not actively using.&lt;/p&gt;

&lt;p&gt;In general, it&apos;s not surprising that auth would change the cost of these kind of pool evictions (as it adds a large fixed overhead to connection establishment, by design).  One pattern you could be in might be:&lt;/p&gt;

&lt;p&gt;All of this in mongos:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Burst of connections to a host&lt;/li&gt;
	&lt;li&gt;no activity&lt;/li&gt;
	&lt;li&gt;connections drop down to minconnections (1) after 60 seconds (the point at which we have to start heart beating a node to keep the connections live)&lt;/li&gt;
	&lt;li&gt;connections drop to 0 after another 4 minutes&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25027&quot; title=&quot;Configurable connection pools size for mongos&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25027&quot;&gt;&lt;del&gt;SERVER-25027&lt;/del&gt;&lt;/a&gt; adds these parameters as exported server params, and we intend to backport these knobs to 3.2.  That would offer one possible way forward (tuning up all the various timeouts) by trading off holding more open connections for longer in return for smoothing connection spikes.&lt;/p&gt;

&lt;p&gt;In the meanwhile, your best mitigation would be to keep some level of regular traffic going to those hosts.&lt;/p&gt;


&lt;p&gt;Regards,&lt;br/&gt;
Jason&lt;/p&gt;</comment>
                            <comment id="1427486" author="sunguck.lee@gmail.com" created="Mon, 7 Nov 2016 01:42:30 +0000"  >&lt;p&gt;Hi Thomas.&lt;/p&gt;

&lt;p&gt;I uploaded mongod and mongos log files (Not for all mongos and mongod, only one mongos and mongod server&apos;s).&lt;/p&gt;

&lt;p&gt;18:30 ~ 19:00 (+09:00)  :: auth enabled test&lt;br/&gt;
20:00 ~ 20:30 (+09:00)  :: auth disabled test&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Matt.&lt;/p&gt;</comment>
                            <comment id="1426370" author="thomas.schubert" created="Fri, 4 Nov 2016 16:18:01 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure upload portal for you to use &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/245cee59-e0be-4984-ab50-9a4a479e9fa5.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. Files uploaded to this portal are visible only to MongoDB employees investigating this issue and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Would you please let us know when you have completed the upload so we can continue to investigate this issue? We&apos;ll continue to investigate this issue here to determine whether there is an appropriate code change that would improve the behavior you are observing.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1425924" author="sunguck.lee@gmail.com" created="Fri, 4 Nov 2016 05:15:01 +0000"  >&lt;p&gt;Andreas Nilsson,&lt;/p&gt;

&lt;p&gt;Is there any way to send log files to you by email or some other ways.&lt;br/&gt;
Log files are too big to attach this thread. (And also don&apos;t want to be public)&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Matt.&lt;/p&gt;</comment>
                            <comment id="1425147" author="andreas.nilsson@10gen.com" created="Thu, 3 Nov 2016 14:33:50 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt; do you have any logs from mongos/mongod during the time of spiky connections? It&apos;s hard to say anything using only the diagnostics data.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Andreas&lt;/p&gt;</comment>
                            <comment id="1424639" author="andreas.nilsson@10gen.com" created="Wed, 2 Nov 2016 21:54:08 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, sorry for not getting to this earlier. It is on our queue and we will look at the metrics file tomorrow.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Andreas&lt;/p&gt;</comment>
                            <comment id="1423775" author="sunguck.lee@gmail.com" created="Wed, 2 Nov 2016 11:45:31 +0000"  >&lt;p&gt;Hi...&lt;/p&gt;

&lt;p&gt;Is there anyone who look into this case.&lt;br/&gt;
Sorry for pushing you guys, But it&apos;s kind of urgent for me.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Matt.&lt;/p&gt;</comment>
                            <comment id="1416368" author="sunguck.lee@gmail.com" created="Tue, 25 Oct 2016 01:42:25 +0000"  >&lt;p&gt;We have restarted a few times during test. &lt;br/&gt;
I think the metrics data you want is stored in &quot;metrics.2016-10-24T10-09-08Z-00000&quot; files.&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</comment>
                            <comment id="1415548" author="ramon.fernandez" created="Mon, 24 Oct 2016 13:09:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.lee&quot; class=&quot;user-hover&quot; rel=&quot;matt.lee&quot;&gt;matt.lee&lt;/a&gt;, can you please upload the logs of a &lt;tt&gt;mongos&lt;/tt&gt; and a &lt;tt&gt;mongod&lt;/tt&gt; during a period of spiky connections? I&apos;d also like to take a look at the contents of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; from a &lt;tt&gt;mongod&lt;/tt&gt; inside one of the shards.&lt;/p&gt;

&lt;p&gt;I&apos;m looking to see where those spiky connections come from, so please choose a representative &lt;tt&gt;mongod&lt;/tt&gt;; for example, if you&apos;re doing secondary reads then include logs and diagnostic data from the node that&apos;s serving those reads as well as its primary.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="330481">SERVER-26952</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="300626">SERVER-25027</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="142175" name="Connections.png" size="38555" author="matt.lee" created="Mon, 24 Oct 2016 12:20:40 +0000"/>
                            <attachment id="142176" name="Query.png" size="46549" author="matt.lee" created="Mon, 24 Oct 2016 12:20:40 +0000"/>
                            <attachment id="142265" name="diagnostic.data.tar.gz" size="11382226" author="matt.lee" created="Tue, 25 Oct 2016 01:42:25 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000XU2hPIAT]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 24 Oct 2016 13:09:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 35 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 35 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andreas.nilsson</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>mira.carey@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer.jackson@mongodb.com</customfieldvalue>
            <customfieldvalue>sunguck.lee@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjssv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hray33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsef6v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>