<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:47:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-59345] Performance degradation after mongo sharded cluster upgrade from 4.2 to 4.4</title>
                <link>https://jira.mongodb.org/browse/SERVER-59345</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi All, We upgraded our sharded cluster a few days back from 4.2 to 4.4, After upgrading to 4.4, our queries are taking much longer and performance continues to deteriorate. Our sharded cluster configuration:&lt;br/&gt;
Instance type: M5.xlarge (data nodes), t3.medium (Config servers)&lt;/p&gt;

&lt;p&gt;Shards: 9&lt;/p&gt;

&lt;p&gt;Replica set type: PSA&lt;/p&gt;

&lt;p&gt;Config Servers: 3&lt;/p&gt;

&lt;p&gt;Mongos&apos;s: 21&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;One thing we noticed is when there are less number of mongos&apos;s, the performance is much better, but enabling more mongos&apos;s leading to performance deterioration.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;We did notice this issue has some similarity to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-51104&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-51104&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks for your help in advance&lt;/p&gt;</description>
                <environment></environment>
        <key id="1851441">SERVER-59345</key>
            <summary>Performance degradation after mongo sharded cluster upgrade from 4.2 to 4.4</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="edwin.zhou@mongodb.com">Edwin Zhou</assignee>
                                    <reporter username="stephenpaul2727@gmail.com">Stephen Paul Adithela</reporter>
                        <labels>
                    </labels>
                <created>Sat, 14 Aug 2021 22:58:28 +0000</created>
                <updated>Thu, 12 May 2022 21:05:41 +0000</updated>
                            <resolved>Tue, 7 Sep 2021 13:48:06 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="4546417" author="JIRAUSER1259436" created="Thu, 12 May 2022 21:05:41 +0000"  >&lt;p&gt;Thank you Stephen for the quick response. Luckily we were able to identify the root cause quick is issue was causing because of hedge reads, had to disable them.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-mongos-hedged-reads&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-mongos-hedged-reads&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4545883" author="stephenpaul2727@gmail.com" created="Thu, 12 May 2022 18:11:01 +0000"  >&lt;p&gt;Hello, In our case the issue has nothing to do with the mongo upgrade as we initially thought. It was due to some degradation on a particular shard, it can create a bottleneck and the performance of the whole DB could go down. We restarted problematic nodes on new hosts (increased WTSE cache+resources temporarily) and that stabilized the env.&lt;/p&gt;</comment>
                            <comment id="4544952" author="JIRAUSER1259436" created="Thu, 12 May 2022 14:29:31 +0000"  >&lt;p&gt;@stephen_paul_adithela - Could you help us know what was the fix performed ?. It would benefit others. Having similar issue where after upgrade memory usage gone down and performance has degraded.&#160;&lt;/p&gt;

&lt;p&gt;Appreciate the help on this.&lt;/p&gt;</comment>
                            <comment id="4042230" author="JIRAUSER1257066" created="Tue, 7 Sep 2021 13:48:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for following up that you were able to resolve this issue. We have had previous tickets (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-57249&quot; title=&quot;Performance degrade after 4.4.3 up-gradation from 4.2 version&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-57249&quot;&gt;&lt;del&gt;SERVER-57249&lt;/del&gt;&lt;/a&gt;) that identified a regression between 4.2 and 4.4 without resolve, so we remain interested in the behavior that caused this regression, and the details of the investigation that took place to help resolve this issue.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4040176" author="stephenpaul2727@gmail.com" created="Fri, 3 Sep 2021 23:59:26 +0000"  >&lt;p&gt;Hi Edwin, Sorry for the late update. We had a consultant from mongo look into our production systems last week. The performance degradation and issues we noticed are due to bottlenecks with wired tiger cache on specific shards. Mongo sharded cluster with 4.4 currently is running fine.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Please resolve this ticket and thanks&lt;/p&gt;</comment>
                            <comment id="4039250" author="JIRAUSER1257066" created="Fri, 3 Sep 2021 15:48:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We still need additional information to diagnose the problem. If this is still an issue for you, would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and attach it to this ticket?&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4005359" author="JIRAUSER1257066" created="Tue, 17 Aug 2021 21:08:42 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for your report.&lt;/p&gt;

&lt;p&gt;Would you please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and attach it to this ticket?&lt;/p&gt;

&lt;p&gt;Can you also let us know the date, time and timezone of when this upgrade was performed?&lt;br/&gt;
Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Aug 2021 21:08:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 38 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ilanchezhianm@gmail.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 38 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>ilanchezhianm@gmail.com</customfieldvalue>
            <customfieldvalue>stephenpaul2727@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzxra7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzhvof:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzxdj3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>