<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:19:57 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29120] mongos connections increase indefinitely on chunk md refresh</title>
                <link>https://jira.mongodb.org/browse/SERVER-29120</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a pretty intensive process that stresses our mongod servers. When this process is running, we encounter the following issue: suddenly, there&apos;s a sudden increase in # of connections mongos opens, and then it starts taking more and more cpu until it becomes totally stuck. Please see the 2 graphs attached.&lt;/p&gt;

&lt;p&gt;When this happens, mongos logs shows A LOT of open/close connection messages (to the mongod servers). This trigger for this process seems to be a chunk move. When the move is complete, the ChunkManager refreshes - and then the connections starts opening and closing endlessly.&lt;/p&gt;

&lt;p&gt;Note that our mongod+mongos run on super strong machines, c4.4xlarge.&lt;/p&gt;

&lt;p&gt;Is this known / any workaround?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22611&quot; title=&quot;ChunkManager refresh can occasionally cause a full reload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22611&quot;&gt;&lt;del&gt;SERVER-22611&lt;/del&gt;&lt;/a&gt; (opened by myself) seems to be related, but fixed only in 3.4, we are using 3.2.12.&lt;/p&gt;</description>
                <environment></environment>
        <key id="382454">SERVER-29120</key>
            <summary>mongos connections increase indefinitely on chunk md refresh</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="yonido">Yoni Douek</reporter>
                        <labels>
                    </labels>
                <created>Wed, 10 May 2017 13:43:50 +0000</created>
                <updated>Mon, 29 Jan 2018 18:34:40 +0000</updated>
                            <resolved>Fri, 16 Jun 2017 21:56:55 +0000</resolved>
                                    <version>3.2.12</version>
                                                    <component>Networking</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1599763" author="kaloian.manassiev" created="Fri, 16 Jun 2017 21:56:55 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoni%40appsee.com&quot; class=&quot;user-hover&quot; rel=&quot;yoni@appsee.com&quot;&gt;yoni@appsee.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Since we haven&apos;t heard back I am going to close this ticket &apos;Cannot Reproduce&apos;. If you are able to get the logs, please reopen it.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1572738" author="kaloian.manassiev" created="Tue, 16 May 2017 20:26:02 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoni%40appsee.com&quot; class=&quot;user-hover&quot; rel=&quot;yoni@appsee.com&quot;&gt;yoni@appsee.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Are you continuing to experience this problem and were you able to obtain the mongos logs? Unfortunately without the logs we have no idea what is going on and even with 3.2.12 logs we might not have sufficient information, but I would like to take a look first.&lt;/p&gt;

&lt;p&gt;I am fairly certain it is not related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22611&quot; title=&quot;ChunkManager refresh can occasionally cause a full reload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22611&quot;&gt;&lt;del&gt;SERVER-22611&lt;/del&gt;&lt;/a&gt;, because this bug manifests as a stall on the mongos side due to repeated full collection reload under a mutex and there is no way this would cause connections to be closed.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1569130" author="yonido" created="Thu, 11 May 2017 17:30:35 +0000"  >&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Random mongos.&lt;/li&gt;
	&lt;li&gt;Ends only on restart.&lt;/li&gt;
	&lt;li&gt;I think it happens AFTER the refresh. The critical section is pretty short so it&apos;s hard to know.&lt;/li&gt;
	&lt;li&gt;5 shards.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Re logs - this time it&apos;s not so easy but ill try to get them. You basically see a lot of connection open/closes. We won&apos;t be able to upgrade to 2.3.13 so easily.&lt;/p&gt;</comment>
                            <comment id="1569017" author="kaloian.manassiev" created="Thu, 11 May 2017 14:55:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoni%40appsee.com&quot; class=&quot;user-hover&quot; rel=&quot;yoni@appsee.com&quot;&gt;yoni@appsee.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would it be possible to attach the log from the mongos which exhibits such an event? Ramon included a secure upload link.&lt;/p&gt;

&lt;p&gt;Also I have a couple of questions:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Is it always the same mongos to which this is happening?&lt;/li&gt;
	&lt;li&gt;Did this event end on its own or you had to restart the mongos?&lt;/li&gt;
	&lt;li&gt;When you say that the trigger for this event is the refresh of the chunk manager - do you mean that the connections start to accumulate after the refresh happens or possibly while the shard is in the critical section and operations are blocked on it?&lt;/li&gt;
	&lt;li&gt;How many shards do you have?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1568469" author="ramon.fernandez" created="Wed, 10 May 2017 16:12:49 +0000"  >&lt;p&gt;I&apos;ve preemptively created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/41a49d2c-d8d7-4dd3-bf3c-97222f7ea184.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; for file uploads. I think we&apos;d need to see logs to determine if this is &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22611&quot; title=&quot;ChunkManager refresh can occasionally cause a full reload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22611&quot;&gt;&lt;del&gt;SERVER-22611&lt;/del&gt;&lt;/a&gt;, but I&apos;m not sure at the moment if 3.2.12 logs will have that information or what logs will be most interesting.&lt;/p&gt;

&lt;p&gt;In 3.2.13 we&apos;ve added a number of logging improvements to help troubleshoot connection related issues, so we may need to ask you to upgrade a &lt;tt&gt;mongos&lt;/tt&gt; to 3.2.13 and provide logs from that node when the behavior you describe appears.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="155725" name="dan1.png" size="123832" author="yonido" created="Wed, 10 May 2017 13:43:31 +0000"/>
                            <attachment id="155726" name="dan2.png" size="99440" author="yonido" created="Wed, 10 May 2017 13:43:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 10 May 2017 16:12:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 34 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 34 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>yonido</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht78pr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hszf7r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrrc7b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>