<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:29:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32182] Deadlock in wiredtiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-32182</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The symptoms of the issue are:&lt;br/&gt;
1. One of the secondary nodes in MongoDB replicaset stops responding on all read requests. Oplog application is also stopped on this node.&lt;br/&gt;
2. Other nodes in replicaset view that failed node as healthy because it responds on pings and status requests.&lt;br/&gt;
3. Investigation shows that all requests on hanged server are waiting for GlobalLock.&lt;/p&gt;</description>
                <environment>Ubuntu 16.04.2 LTS, x86&lt;br/&gt;
AWS EC2 i3 instance</environment>
        <key id="468644">SERVER-32182</key>
            <summary>Deadlock in wiredtiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="keith.bostic@mongodb.com">Keith Bostic</assignee>
                                    <reporter username="sz">Sergey Zagursky</reporter>
                        <labels>
                    </labels>
                <created>Wed, 6 Dec 2017 08:45:58 +0000</created>
                <updated>Mon, 29 Jan 2018 18:34:53 +0000</updated>
                            <resolved>Fri, 12 Jan 2018 19:06:30 +0000</resolved>
                                    <version>3.4.6</version>
                                                    <component>WiredTiger</component>
                                        <votes>1</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="1773612" author="sz" created="Sat, 13 Jan 2018 07:12:09 +0000"  >&lt;p&gt;I haven&apos;t encountered this issue any more. We&apos;ve upgraded MongoDB to 3.4.10 shortly after reporting the issue here.&lt;/p&gt;</comment>
                            <comment id="1773150" author="keith.bostic" created="Fri, 12 Jan 2018 19:05:56 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;, I&apos;m going to close this ticket for now: I can&apos;t think of any way to pursue this problem without additional information and I&apos;ve been unable to reproduce the failure in any test. Please don&apos;t hesitate to re-open this ticket or open a new one if there&apos;s any additional information or further problems.&lt;/p&gt;</comment>
                            <comment id="1765449" author="keith.bostic" created="Thu, 4 Jan 2018 20:50:24 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;, I was wondering if you&apos;ve seen this failure in the last month?&lt;/p&gt;

&lt;p&gt;And while reviewing the ticket, I realized we hadn&apos;t asked you if the failure was on the same piece of hardware (and if so, if that hardware is still running?)&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="1754688" author="sz" created="Mon, 18 Dec 2017 17:30:01 +0000"  >&lt;blockquote&gt;&lt;p&gt;Has the deadlock occurred again, since you filed the ticket?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;No.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;If this is reproducible for you in a reasonable amount of time, the next step might be to provide you an instrumented build for you to run (but that build might quite possibly have different performance characteristics than the standard builds). Would that be possible and worth the additional effort for you?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Unfortunately, it isn&apos;t reproducible consistently enough. It occured three times total, 11 Nov, 02 Dec and 04 Dec. There is no pattern I&apos;m aware of. 02 Dec and 04 Dec definitely weren&apos;t under heavy load. In fact the load was pretty low.&lt;br/&gt;
As this is production system, we are somewhat limited performance-wise. We definitely can&apos;t allow x2 throughput reduce. 10-20% performance penalty should be OK. What will performance penalty be?&lt;/p&gt;

&lt;p&gt;I still have core dump here. Can I be of any help inspecting it? I can be your hands and eyes &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1754597" author="keith.bostic" created="Mon, 18 Dec 2017 16:40:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;, I&apos;m afraid we&apos;re stuck on this one: we&apos;ve reviewed the information and run experiments trying to reproduce the failure, all without success.&lt;/p&gt;

&lt;p&gt;There was the one problem we&apos;ve documented running on Azure, but that&apos;s the only explanation that we have, and the problem isn&apos;t happening anywhere else as far as we know.&lt;/p&gt;

&lt;p&gt;Has the deadlock occurred again, since you filed the ticket?&lt;/p&gt;

&lt;p&gt;If this is reproducible for you in a reasonable amount of time, the next step might be to provide you an instrumented build for you to run (but that build might quite possibly have different performance characteristics than the standard builds). Would that be possible and worth the additional effort for you?&lt;/p&gt;

&lt;p&gt;Otherwise, I don&apos;t see any way to make progress on this one.&lt;/p&gt;

&lt;p&gt;I&apos;m truly sorry for the inconvenience, I wish it were otherwise!&lt;/p&gt;</comment>
                            <comment id="1745298" author="sz" created="Thu, 7 Dec 2017 13:55:24 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=keith.bostic&quot; class=&quot;user-hover&quot; rel=&quot;keith.bostic&quot;&gt;keith.bostic&lt;/a&gt;&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;How often have you seen this problem, does it repeat, or was it just the one time?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;We&apos;ve seen this problem three times so far.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;And, how long did you wait for the secondary to wake up?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The wait time was different each time. But it definitely wasn&apos;t on the scale of seconds. IIRC the shortest was 10 minutes. The longest was almost 50 minutes.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;There is a possibly related problem one of our developers flagged (&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3461&quot; title=&quot;Avoid long sleeps when the system clock is adjusted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3461&quot;&gt;&lt;del&gt;WT-3461&lt;/del&gt;&lt;/a&gt;/&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31215&quot; title=&quot;Enable pthread cond monotonic on Linux for WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31215&quot;&gt;&lt;del&gt;SERVER-31215&lt;/del&gt;&lt;/a&gt;), but the fix for that problem isn&apos;t yet included in a MongoDB 3.4 release (it will be included in the 3.4.11 release).&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I personally don&apos;t think our issue is related to &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3461&quot; title=&quot;Avoid long sleeps when the system clock is adjusted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3461&quot;&gt;&lt;del&gt;WT-3461&lt;/del&gt;&lt;/a&gt;/&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31215&quot; title=&quot;Enable pthread cond monotonic on Linux for WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31215&quot;&gt;&lt;del&gt;SERVER-31215&lt;/del&gt;&lt;/a&gt; because system time was pretty stable at the moment of failure. There also were no manual time adjustements at that time. Timestamps in mongod.log are seemingly consequent. Although we can&apos;t exclude skew possibility for sure, because we have no time monitoring on failed instances.&lt;/p&gt;</comment>
                            <comment id="1745269" author="keith.bostic" created="Thu, 7 Dec 2017 12:40:11 +0000"  >&lt;p&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;, a couple of questions:&lt;/p&gt;

&lt;p&gt;How often have you seen this problem, does it repeat, or was it just the one time?&lt;/p&gt;

&lt;p&gt;And, how long did you wait for the secondary to wake up?&lt;/p&gt;

&lt;p&gt;There is a possibly related problem one of our developers flagged (&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3461&quot; title=&quot;Avoid long sleeps when the system clock is adjusted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3461&quot;&gt;&lt;del&gt;WT-3461&lt;/del&gt;&lt;/a&gt;/&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31215&quot; title=&quot;Enable pthread cond monotonic on Linux for WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31215&quot;&gt;&lt;del&gt;SERVER-31215&lt;/del&gt;&lt;/a&gt;), but the fix for that problem isn&apos;t yet included in a MongoDB 3.4 release (it will be included in the 3.4.11 release).  However, while that problem can occur on Linux systems, it has been generally been seen on Windows systems, specifically on Azure.&lt;/p&gt;</comment>
                            <comment id="1744560" author="ramon.fernandez" created="Wed, 6 Dec 2017 19:25:56 +0000"  >&lt;p&gt;Thanks for the detailed report &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;, we&apos;re looking at the stack traces you provided.&lt;/p&gt;
</comment>
                            <comment id="1743915" author="sz" created="Wed, 6 Dec 2017 08:48:21 +0000"  >&lt;p&gt;I&apos;ve attached stack traces of all mongod threads at the moment of deadlock.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="416166">WT-3504</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="172484" name="stacktrace.txt" size="184950" author="sz" created="Wed, 6 Dec 2017 08:46:51 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 6 Dec 2017 14:02:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 4 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 4 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>keith.bostic@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>sz</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htlo93:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htd9bj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2094">Storage 2018-01-29</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htlad3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>