<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:52:48 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39684] Mongod Server slows down and caused high CPU usage</title>
                <link>https://jira.mongodb.org/browse/SERVER-39684</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Dear All,&lt;/p&gt;

&lt;p&gt;We have a Replica Set with 12 nodes total, 6 nodes v3.2.21, 6 nodes v3.2.22, and 1 hidden member. We have 40k connections from our clients, and we cannot reduce this number at the moment.&lt;/p&gt;

&lt;p&gt;The problem is at some time of the day (usually in the early morning), mongod will become very slow, even isMaster command will takes ~200 ms. If we check our Kibana slow logs, most queries have high timeAcquiringMicros. If we does not restart the mongod, it will result in unreachable secondary, high CPU load and Context Switches, and crashed the server. Other metrics like Memory, Disk Usage are normal during the mongod slowdown. We checked slow logs and error logs, but we haven&apos;t found any suspicious queries that can cause this, so our suspect is mongod becomes slow after there are some lock contention. We still do not know what is the root cause of this lock contention though.&lt;/p&gt;

&lt;p&gt;This happens almost everyday, we restart one or two nodes every morning because of high CPU and server crash. We would like to know if we could see what kind of locks are frequently used, or what are the possible causes of this kind of behavior.&#160;&lt;/p&gt;

&lt;p&gt;Any help would be appreciated.&lt;br/&gt;
Thank you.&lt;/p&gt;

&lt;p&gt;PS : We still use 2.6 credentials, does it matter?&lt;/p&gt;</description>
                <environment>Debian 8</environment>
        <key id="701107">SERVER-39684</key>
            <summary>Mongod Server slows down and caused high CPU usage</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="eric.sedor@mongodb.com">Eric Sedor</assignee>
                                    <reporter username="kevinsupendi">Kevin Supendi</reporter>
                        <labels>
                    </labels>
                <created>Wed, 20 Feb 2019 08:20:24 +0000</created>
                <updated>Tue, 21 May 2019 03:53:15 +0000</updated>
                            <resolved>Thu, 21 Feb 2019 18:00:32 +0000</resolved>
                                    <version>3.2.21</version>
                    <version>3.2.22</version>
                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="2253664" author="evan.z" created="Tue, 21 May 2019 03:53:15 +0000"  >&lt;p&gt;Hello&#160; Kevin ,&#160;I have recently encountered the same problem. We can only monitor the logs every day.&#160;When &apos;the server is very slow&apos; appears, we restart mongod.&#160;Now the frequency is about once a week.Have you solved this problem?&#160;Any help would be appreciated.&lt;br/&gt;
Thank you.&lt;/p&gt;</comment>
                            <comment id="2158603" author="eric.sedor" created="Thu, 21 Feb 2019 17:29:24 +0000"  >&lt;p&gt;My apologizes Kevin, I overlooked the MongoDB version you provided initially.&lt;/p&gt;

&lt;p&gt;MongoDB 3.2 reached end of life in September of 2018 and the SERVER project is for bugs and feature requests for current versions. We aren&apos;t able to help you diagnose this deployment here.&lt;/p&gt;

&lt;p&gt;For MongoDB-related support discussion please post on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-user group.&lt;/p&gt;

&lt;p&gt;See also our &lt;a href=&quot;https://docs.mongodb.org/manual/support&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Technical Support page&lt;/a&gt; for additional support resources.&lt;/p&gt;</comment>
                            <comment id="2158082" author="kevinsupendi" created="Thu, 21 Feb 2019 07:38:20 +0000"  >&lt;p&gt;Oh i want to add, we set Db profiling level to 1, and slow log threshold 200 ms&lt;/p&gt;</comment>
                            <comment id="2158078" author="kevinsupendi" created="Thu, 21 Feb 2019 07:17:11 +0000"  >&lt;p&gt;Hello Eric, thank you for your response. Yesterday, it happened again, 5 nodes were down. Let&apos;s call them node A (primary), B, C, D and E. They went down approximately between&#160;20 Feb 2019 05:30:00 to&#160;06:00:00 GMT+7.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I couldn&apos;t attach their files individually because they exceed 150 MB file upload limit, so I put the zip (1.1GB) in this GDrive link.&#160;&lt;a href=&quot;https://drive.google.com/file/d/1YyyzSyQWFJdTAvkspnEBK0XUh-4m-mWl/view?usp=sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://drive.google.com/file/d/1YyyzSyQWFJdTAvkspnEBK0XUh-4m-mWl/view?usp=sharing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The zip file consists of 5 diagnostic.data zip for each server and a day of mongod logs for each server. We hope that you can help us pinpoint the problem in our production setup.&lt;/p&gt;

&lt;p&gt;Thank you for your time.&lt;/p&gt;</comment>
                            <comment id="2157931" author="eric.sedor" created="Thu, 21 Feb 2019 01:19:32 +0000"  >&lt;p&gt;Hello, would you please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory and attach it to this ticket? A day of server logs that cover one or two incidents will also be helpful.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="209527" name="kibana-log.PNG" size="75311" author="kevinsupendi" created="Wed, 20 Feb 2019 08:06:03 +0000"/>
                            <attachment id="209526" name="mongocpu.PNG" size="78551" author="kevinsupendi" created="Wed, 20 Feb 2019 08:12:51 +0000"/>
                            <attachment id="209525" name="mongomemory.PNG" size="61159" author="kevinsupendi" created="Wed, 20 Feb 2019 08:12:51 +0000"/>
                            <attachment id="209524" name="mongooverview.PNG" size="104569" author="kevinsupendi" created="Wed, 20 Feb 2019 08:12:51 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 21 Feb 2019 01:19:32 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 38 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>evan.z</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 38 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>evan.z</customfieldvalue>
            <customfieldvalue>kevinsupendi</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huo4nb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hudvjz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunqwn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>