<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:49:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-82398] Excessive memory and CPU consumption during normal operation</title>
                <link>https://jira.mongodb.org/browse/SERVER-82398</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi!&lt;/p&gt;

&lt;p&gt;On one of our shards at some point in time the consumed memory of the primary replica started to rapidly grow along with high CPU consumption. Then that replica became unresponsive, and consequently another replica became the primary. Right after that the same happened to the new primary.&lt;/p&gt;

&lt;p&gt;The incident timeline:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;10/24/23 7:40 - beginning (peak in CPU and memory consumption)&lt;/li&gt;
	&lt;li&gt;10/24/23 8:20-8:26 (can&apos;t say exact time) - the primary (replica-1) becomes unresponsive, another replica (replica-2) becomes the new primary, and we see peak in CPU and memory consumption again&lt;/li&gt;
	&lt;li&gt;10/24/23 8:38 - the new primary (replica-2) becomes unresponsive, another replica (replica-1) becomes the new primary&lt;/li&gt;
	&lt;li&gt;10/24/23 8:43 - the replica (replica-3) that didn&apos;t appear to ever assume the primary role starts experience the same problems with CPU and memory&lt;/li&gt;
	&lt;li&gt;10/24/23 9:20 - we manually restart replica-3, the incident ends&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Unfortunately, we couldn&apos;t get to the core of the problem, but here some things we could observe:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;we noticed that the amount of open cursors jumped up to 500 at the replicas mentioned above (we use Change Streams, so it might be related)&lt;/li&gt;
	&lt;li&gt;On replica-3 there were dozens of &quot;hanging&quot; aggregation commands (in secs_running we saw pretty big numbers, like 2000 seconds)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Could you help us identify the cause of the problem?&lt;/p&gt;

&lt;p&gt;I&apos;m attaching the diagnosting data of the aforementioned replicas (I named the files with replica-1, replica-2 and replica-3, these names correspond to the replica numbers mentioned above).&lt;/p&gt;</description>
                <environment>Ubuntu 18.04.6 LTS&lt;br/&gt;
XSF&lt;br/&gt;
Kernel - 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux&lt;br/&gt;
Disable Transparent Huge disabled&lt;br/&gt;
AWS m5.2xlarge&lt;br/&gt;
SSD GP3 450 Gb</environment>
        <key id="2481774">SERVER-82398</key>
            <summary>Excessive memory and CPU consumption during normal operation</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="11262" iconUrl="https://jira.mongodb.org/images/icons/statuses/generic.png" description="">Investigating</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="yuan.fang@mongodb.com">Yuan Fang</assignee>
                                    <reporter username="vladimirred456@gmail.com">Vladimir Beliakov</reporter>
                        <labels>
                    </labels>
                <created>Tue, 24 Oct 2023 13:24:17 +0000</created>
                <updated>Mon, 1 Jan 2024 08:01:05 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5952484" author="JIRAUSER1270794" created="Thu, 14 Dec 2023 23:04:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your report and for providing us with the FTDC. When looking at the FTDC of replica-1 (primary), starting from &lt;b&gt;A&lt;/b&gt; (2023-10-23T07:40:06.497Z):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/497227/497227_SERVER-82398.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;There is an increase in CPU utilization, escalating cache pressure (&lt;tt&gt;cache fill ratio&lt;/tt&gt; exceeding 80%), and a rise in cache eviction (&lt;tt&gt;ss wt cache pages queued for eviction&lt;/tt&gt;, &lt;tt&gt;ss wt cache unmodified pages evicted&lt;/tt&gt;). The read tickets become exhausted, and it appears they are waiting for the cache to become available (&lt;tt&gt;ss wt thread-yield page acquire time sleeping&lt;/tt&gt;).&lt;/p&gt;

&lt;p&gt;The readers are queued up, and, as you also mentioned, the &lt;tt&gt;ss metrics cursor open pinned&lt;/tt&gt; increased to 415, indicating active use by running queries. I suspect there may be a read workload change at A ( even though there is not a clear event seen from &lt;tt&gt;ss opcounters query&lt;/tt&gt;), however, there are many spikes in &lt;tt&gt;ss metrics queryExecutor scanned&lt;/tt&gt; after A. Do you know which queries are responsible for this??&lt;/p&gt;

&lt;p&gt;I recommend gaining a better understanding of the workload during the incident. Could you please review the logs and check for any interesting slow queries on replica-1 in the period of A-B? &lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Yuan&lt;/p&gt;</comment>
                            <comment id="5804940" author="vladimirred456@gmail.com" created="Wed, 25 Oct 2023 13:17:05 +0000"  >&lt;p&gt;I forgot to mention that our MongoDB version is 5.0.14&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="484946" name="1 - replica-1 (was primary before the incident happened).zip" size="36495198" author="vladimirred456@gmail.com" created="Tue, 24 Oct 2023 13:24:05 +0000"/>
                            <attachment id="484947" name="2 - replica-2 (became primary during the incident).zip" size="53441777" author="vladimirred456@gmail.com" created="Tue, 24 Oct 2023 13:24:09 +0000"/>
                            <attachment id="484945" name="3 - replica-3 (didn&apos;t notice that it was primary, but had same problems).zip" size="33637231" author="vladimirred456@gmail.com" created="Tue, 24 Oct 2023 13:24:03 +0000"/>
                            <attachment id="497227" name="SERVER-82398.png" size="536069" author="yuan.fang@mongodb.com" created="Thu, 14 Dec 2023 23:00:39 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25137"><![CDATA[Server Triage]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 14 Dec 2023 23:04:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>vladimirred456@gmail.com</customfieldvalue>
            <customfieldvalue>yuan.fang@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2xbqn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2f72k:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[yuan.fang@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2wxvz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>