<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:16:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-48310] Primary node GlobalLock suddenly arise making service unavailable</title>
                <link>https://jira.mongodb.org/browse/SERVER-48310</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&#160;&lt;/p&gt;

&lt;p&gt;We have a hard to identify issue here needs your help. One instance&apos;s primary node suffered from GlobalLock suddenly arising, consuming out of all the connections. It looks like relative with WiredTiger cache evict problem because at that time the cache usage had reached 95%. This issue happened at 2020-05-18:12:36:45 GMT+8, and I have provided the FTDC data and some logs needed. Please help look into this issue. Thanks!&lt;/p&gt;</description>
                <environment></environment>
        <key id="1356639">SERVER-48310</key>
            <summary>Primary node GlobalLock suddenly arise making service unavailable</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13204">Community Answered</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="mingyan.zc@gmail.com">Cen Zheng</reporter>
                        <labels>
                    </labels>
                <created>Wed, 20 May 2020 01:55:17 +0000</created>
                <updated>Fri, 27 Oct 2023 15:56:34 +0000</updated>
                            <resolved>Tue, 7 Jul 2020 07:31:19 +0000</resolved>
                                    <version>4.0.3</version>
                                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="3271564" author="dmitry.agranat" created="Tue, 7 Jul 2020 07:31:05 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mingyan.zc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;mingyan.zc@gmail.com&quot;&gt;mingyan.zc@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3133169" author="mingyan.zc@gmail.com" created="Thu, 28 May 2020 01:51:59 +0000"  >&lt;p&gt;Hi Carl,&lt;/p&gt;

&lt;p&gt;Ok. I will try turn off the profiler and see if it will still happen. Thanks!&lt;/p&gt;</comment>
                            <comment id="3120112" author="carl.champain" created="Wed, 27 May 2020 14:45:43 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mingyan.zc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;mingyan.zc@gmail.com&quot;&gt;mingyan.zc@gmail.com&lt;/a&gt;,&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;I saw the Metadata W lock arising was at 13:30 GMT+8. The issue happened at 12:36:45 GMT+8.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The&#160;&lt;tt&gt;locks Metadata timeAcquiringMicros W&lt;/tt&gt; starts rising at 12:35:44.085 GMT+8 (marker A on the screenshot in my previous comment).&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Yes, we have profiler on by default. But I think this might not be the root cause, just making things worse only.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Have you tried turning the profiler off? We are happy to keep investigating if the problem persists after disabling profiling.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Carl&lt;br/&gt;
&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3119477" author="mingyan.zc@gmail.com" created="Wed, 27 May 2020 01:44:57 +0000"  >&lt;p&gt;Hi, Carl,&lt;/p&gt;

&lt;p&gt;Thanks for your analyst. Yes, we have profiler on by default. But I think this might not be the root cause, just making things worse only. Because when wt cache usage had reached 95%, all application threads needed to help evict so all normal requests became slow. I saw the Metadata W lock arising was at 13:30 GMT+8. The issue happened at&#160;12:36:45 GMT+8.&lt;/p&gt;</comment>
                            <comment id="3107437" author="carl.champain" created="Tue, 26 May 2020 17:44:59 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mingyan.zc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;mingyan.zc@gmail.com&quot;&gt;mingyan.zc@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/262506/262506_screenshot.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;The bottleneck seems to be &lt;tt&gt;locks Metadata timeAcquiringMicros W&lt;/tt&gt; with 177 threads waiting on locks. Please note that this metric involves capped collections and we saw that the profiler is on (the profiler is a capped collection). We&#8217;ve seen similar issues in the past due to having the profiler on. Can you please turn off the profiler and see if it solves your issue?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Carl&lt;/p&gt;

</comment>
                            <comment id="3098711" author="mingyan.zc@gmail.com" created="Thu, 21 May 2020 03:08:46 +0000"  >&lt;p&gt;Hi Carl,&lt;/p&gt;

&lt;p&gt;I have uploaded log file before&#160;2020-05-18:12:36:45 GMT+8, please check.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="3098116" author="carl.champain" created="Wed, 20 May 2020 19:31:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mingyan.zc%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;mingyan.zc@gmail.com&quot;&gt;mingyan.zc@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report.&lt;/p&gt;

&lt;p&gt;The earliest log line in the mongod.log files is at &lt;tt&gt;2020-05-18T13:09:07.762+0800&lt;/tt&gt;. To help us investigate this issue, we would also need the mongod.log files covering this behavior from before it started (so a bit before &lt;tt&gt;2020-05-18:12:36:45 GMT+8&lt;/tt&gt;). Can you please provide them?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
 Carl&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="261710" name="diagnostic.data.tar.gz" size="41645702" author="mingyan.zc@gmail.com" created="Wed, 20 May 2020 01:54:43 +0000"/>
                            <attachment id="261929" name="mongod.log.2020-05-18T05-01-07.gz" size="22898137" author="mingyan.zc@gmail.com" created="Thu, 21 May 2020 03:07:58 +0000"/>
                            <attachment id="261711" name="mongod.log.tar.gz" size="23147383" author="mingyan.zc@gmail.com" created="Wed, 20 May 2020 01:54:30 +0000"/>
                            <attachment id="262506" name="screenshot.png" size="139820" author="carl.champain@mongodb.com" created="Tue, 26 May 2020 17:42:46 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 20 May 2020 19:31:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 31 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 31 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>carl.champain@mongodb.com</customfieldvalue>
            <customfieldvalue>mingyan.zc@gmail.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxm2y7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx9c3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxlp7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>