<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:47:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-38074] abnormal performance downgrade and large traffice between primary shard and config server</title>
                <link>https://jira.mongodb.org/browse/SERVER-38074</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Mongo Version 3.2.20, clusters deployed as: 15 or more shard replica sets + 1 config replica sets.&lt;/p&gt;

&lt;p&gt;Each shard has three nodes(1 primary + 2 secondary), the storage is mmapv1.&lt;/p&gt;

&lt;p&gt;Abnormal performance downgrade occur irregularly:&lt;/p&gt;

&lt;p&gt;1. The traffics output from all config nodes are very large.&lt;br/&gt;
 2. only occurs between the primary node on primary shard and config servers.&lt;br/&gt;
 3. when the situation occurs, we found large request from primary shard to config servers, logs as below:&lt;/p&gt;

&lt;p&gt;log a:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;2018-11-07T18:59:07.289959+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn51755&amp;#93;&lt;/span&gt; remotely refreshing metadata for ${db}.${collection} with requested shard version 0|0||000000000000000000000000, current shard version is 23075|5302||5bd9add9f47203867a329afd, current metadata version is 23081|428||5bd9add9f47203867a329afd&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;log b:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;2018-11-07T22:03:44.271591+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn63943&amp;#93;&lt;/span&gt; updating metadata for ${db}.${collection} from shard version 23108|23||5bd9add9f47203867a329afd to shard version 23108|23||5bd9add9f47203867a329afd, took 850 ms&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;4. at this time, large slow logs were generated on primary node, the op time were growing, db.currentOp() on primary shard, shows that many queries were `&quot;msg&quot; : &quot;waiting for write concern&quot;,`&lt;/p&gt;

&lt;p&gt;5. after we rise the loglevel, we found large FIND QUERY for chunks collection from primary node to config servers.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;2018-11-09T19:03:00.370867+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-ShardRegistry-0&amp;#93;&lt;/span&gt; Initiating asynchronous command: RemoteCommand 608607 &amp;#8211; target:${config svr ip}:{port} db:config expDate:2018-11-09T19:03:30.370+0800 cmd:{ find: &quot;chunks&quot;, filter: { ns: &quot;${db}.${collection}&quot;, lastmod: { $gte: Timestamp 0|0 } }, sort: { lastmod: 1 }, readConcern: { level: &quot;majority&quot;, afterOpTime: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { ts}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt; }, maxTimeMS: 30000 }&lt;br/&gt;
2018-11-09T19:03:00.370883+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-ShardRegistry-0&amp;#93;&lt;/span&gt; Starting asynchronous command 608607 on host ${config server ip}:${port}&lt;br/&gt;
2018-11-09T19:03:00.372395+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-ShardRegistry-0&amp;#93;&lt;/span&gt; Request 608607 finished with response: { waitedMS: 0, cursor: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { firstBatch}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;, ok: 1.0 }&lt;br/&gt;
2018-11-09T19:03:00.373470+08:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-ShardRegistry-0&amp;#93;&lt;/span&gt; Failed to time operation 608607 out: Operation aborted.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;6. This situation can be solved temporarily through stepDown config primary or stopDown primary shard. But it will happen again in the uncertain time of the future.&lt;br/&gt;
 7. after stepDown the primary shard, the slowlogs(UPDATE) on old primay node keep growing, we still need to restart the mongod on old primay nodeto stop it.&lt;br/&gt;
 8. We don&apos;t have the method to reproduce, but it occurs.&lt;/p&gt;</description>
                <environment></environment>
        <key id="632007">SERVER-38074</key>
            <summary>abnormal performance downgrade and large traffice between primary shard and config server</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="stutiredboy@gmail.com">Adun</reporter>
                        <labels>
                    </labels>
                <created>Sat, 10 Nov 2018 06:09:36 +0000</created>
                <updated>Mon, 12 Nov 2018 14:40:50 +0000</updated>
                            <resolved>Mon, 12 Nov 2018 14:02:27 +0000</resolved>
                                    <version>3.2.20</version>
                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2058597" author="stutiredboy@gmail.com" created="Mon, 12 Nov 2018 14:36:45 +0000"  >&lt;p&gt;config servers use WiredTiger as storage engine.&lt;/p&gt;

&lt;p&gt;shard servers use MMAPv1 as storeage engine.&lt;/p&gt;

&lt;p&gt;It occured irregularly.&#160; The cluster are available most of the time.&#160;&lt;/p&gt;</comment>
                            <comment id="2058556" author="daniel.hatcher" created="Mon, 12 Nov 2018 14:02:17 +0000"  >&lt;p&gt;Hello Adun,&lt;/p&gt;

&lt;p&gt;You mention in your initial description that you have config servers deployed in a replica set but you are using MMAPv1 as a storage engine. Please note that when deployed in a replica set, &lt;a href=&quot;https://docs.mongodb.com/v3.2/core/sharded-cluster-config-servers/#replica-set-config-servers&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;config servers must be using the WiredTiger storage engine&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Your current issue may be the result of config server unavailability. &lt;a href=&quot;https://docs.mongodb.com/v3.2/core/sharded-cluster-config-servers/#config-server-availability&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Config servers must be available&lt;/a&gt; at all times in order to ensure the normal operation of a sharded cluster.&lt;/p&gt;

&lt;p&gt;For further MongoDB-related support discussion, please post on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-user group.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 12 Nov 2018 14:02:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 13 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>stutiredboy@gmail.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 13 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>stutiredboy@gmail.com</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huckhz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu2rvr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huc6rb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>