<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:52:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39658] Sudden CPU spike in secondary instances with no apparent cause</title>
                <link>https://jira.mongodb.org/browse/SERVER-39658</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Out of nowhere the two secondary instances for a three node cluster with very little CPU usage spiked to 100% and remained high.&lt;/p&gt;

&lt;p&gt;Version:&#160;2.4.14&lt;/p&gt;

&lt;p&gt;Things we did so far:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Rebuilt all three databases from replicas.&lt;/li&gt;
	&lt;li&gt;reIndex()&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Things we discovered:&lt;/p&gt;

&lt;p&gt;Now that the load is higher and avg response time is &amp;gt;100ms than usual we see some query that should be using index not using it.&lt;/p&gt;

&lt;p&gt;Tue Feb 19 12:47:27.946 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn11958&amp;#93;&lt;/span&gt; query locator.pages query: { query: { _depth: &lt;/p&gt;
{ $gte: 1, $lte: 1 }
&lt;p&gt;, _site: &quot;sample.com&quot; }, orderby: { _id: 1 } } ntoreturn:0 ntoskip:0 nscanned:2106803 keyUpdates:0 numYields: 11 locks(micros) r:5514254 nreturned:30 reslen:130866 2951ms&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209391/209391_image-2019-02-19-04-52-58-249.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Screenshots&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Secondary Node 1 - The other follows the same pattern.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209393/209393_image-2019-02-19-04-44-10-051.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Primary Node (mostly idle except for nightly batch loads)&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209392/209392_image-2019-02-19-04-45-04-009.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="700134">SERVER-39658</key>
            <summary>Sudden CPU spike in secondary instances with no apparent cause</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-triage">Backlog - Triage Team</assignee>
                                    <reporter username="sctrilogy">Santiago Ciciliani</reporter>
                        <labels>
                    </labels>
                <created>Tue, 19 Feb 2019 12:53:31 +0000</created>
                <updated>Tue, 6 Dec 2022 03:05:31 +0000</updated>
                            <resolved>Wed, 20 Feb 2019 00:11:39 +0000</resolved>
                                                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="2156553" author="sctrilogy" created="Wed, 20 Feb 2019 00:56:49 +0000"  >&lt;p&gt;Hi Eric, thanks for your response. I am aware that MongoDB 2.4 is EoL and we are in the process of analyzing upgrade options.&lt;/p&gt;

&lt;p&gt;In the meantime, do you know if there is a workaround we could apply to correct the plan choice and get the avg load back to normal?&lt;/p&gt;

&lt;p&gt;I&apos;m puzzled by the fact that the server restored from the snapshot is choosing the right plan considering that the snapshot was taken after the issue started happening.&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;</comment>
                            <comment id="2156519" author="eric.sedor" created="Wed, 20 Feb 2019 00:10:48 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sctrilogy&quot; class=&quot;user-hover&quot; rel=&quot;sctrilogy&quot;&gt;sctrilogy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We believe you&apos;ve found the likely reason for the CPU use, which is a common symptom of a sudden poor plan choice. Because of how the MongoDB query planner works it is possible for index choice to change. As well, several bugs involving poor plan choice have been corrected since MongoDB 2.4. A recent major improvement was &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20139&quot; title=&quot;Enable CachedPlanStage replanning by default in 3.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20139&quot;&gt;&lt;del&gt;SERVER-20139&lt;/del&gt;&lt;/a&gt; in MongoDB 3.0.&lt;/p&gt;

&lt;p&gt;Unfortunately MongoDB 2.4 reached end of life in March of 2016 and the SERVER project is for bugs or feature suggestions for supported versions of the MongoDB server.&lt;/p&gt;

&lt;p&gt;For MongoDB-related support discussion please post on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-user group.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="2155602" author="sctrilogy" created="Tue, 19 Feb 2019 15:47:35 +0000"  >&lt;p&gt;Further update.&lt;/p&gt;

&lt;p&gt;We cloned the database from AWS AMI images and we run the same query through profiler. It turns out that new-restored cluster seems to be using the index and current prod is doing a sequential scan.&lt;/p&gt;

&lt;p&gt;Any hint on how to fix this?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209415/209415_mongodb-24-query-compare.JPG&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="209393" name="image-2019-02-19-04-44-10-051.png" size="102180" author="sctrilogy" created="Tue, 19 Feb 2019 12:44:12 +0000"/>
                            <attachment id="209392" name="image-2019-02-19-04-45-04-009.png" size="70270" author="sctrilogy" created="Tue, 19 Feb 2019 12:45:05 +0000"/>
                            <attachment id="209391" name="image-2019-02-19-04-52-58-249.png" size="59980" author="sctrilogy" created="Tue, 19 Feb 2019 12:52:59 +0000"/>
                            <attachment id="209415" name="mongodb-24-query-compare.JPG" size="117396" author="sctrilogy" created="Tue, 19 Feb 2019 15:47:11 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25137"><![CDATA[Server Triage]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 20 Feb 2019 00:10:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 51 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 51 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-triage</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>sctrilogy</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunyn3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hudprz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunkwf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>