<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:37:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34726] Deadlock with locally stashed transaction resources during profiling</title>
                <link>https://jira.mongodb.org/browse/SERVER-34726</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;If a &lt;tt&gt;dropDatabase&lt;/tt&gt; is run concurrently with a query operation that&apos;s part of a transaction, a deadlock may occur if the query operation attempts to write profiling information.&lt;/p&gt;

&lt;p&gt;Consider the following situation: There are two threads, T1 and T2. T1 needs to have profiling enabled. Here&apos;s a timeline of what could happen:&lt;/p&gt;

&lt;p&gt;-T1 starts an operation as part of a transaction and acquires its first lock&lt;br/&gt;
-T2 performs a dropDatabase and requests an X lock (though it is not granted yet because T1 holds an IX lock on the collection being dropped)&lt;br/&gt;
-T1&apos;s command execution finishes and &lt;tt&gt;profile()&lt;/tt&gt; is called.&lt;/p&gt;

&lt;p&gt;In &lt;tt&gt;profile()&lt;/tt&gt; the &lt;tt&gt;OperationContext&lt;/tt&gt;&apos;s &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e30163d13fcb1d0a09a09a7569089a61d3551754/src/mongo/db/ops/write_ops_exec.cpp#L130-L131&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transaction resources are stored in a temporary local variable&lt;/a&gt;, and the &lt;tt&gt;OperationContext&lt;/tt&gt; is given a new &lt;tt&gt;RecoveryUnit&lt;/tt&gt; and &lt;tt&gt;Locker&lt;/tt&gt;. With the new Locker, &lt;tt&gt;profile()&lt;/tt&gt; requests an IX lock on the &lt;tt&gt;system.profile&lt;/tt&gt; collection. The new lock request is enqueued behind the dropDatabase&apos;s X lock and is not granted.&lt;/p&gt;

&lt;p&gt;This leaves us with the following lock cycle:&lt;/p&gt;

&lt;p&gt;The lock requested by &lt;tt&gt;profile()&lt;/tt&gt; in T1 cannot be granted until the drop (from T2) has finished.&lt;/p&gt;

&lt;p&gt;The X lock requested by T2 cannot be granted until the locks in the transaction resources stored locally in T1 are released.&lt;/p&gt;

&lt;p&gt;The transaction resources stored in a local variable cannot be moved back to the &lt;tt&gt;OperationContext&lt;/tt&gt; until &lt;tt&gt;profile()&lt;/tt&gt; has completed. Since T1 is deadlocked and its transaction resources are not associated with the session, they cannot be freed (even by the transaction killer or session killer thread).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=james.wahlin&quot; class=&quot;user-hover&quot; rel=&quot;james.wahlin&quot;&gt;james.wahlin&lt;/a&gt; and I believe that one way to fix this would be to stash the TxnResources back on the session instead of in a local variable when &lt;tt&gt;shouldDBProfile&lt;/tt&gt; returns true. This way, the transaction killer or session killer would be able to free the &lt;tt&gt;TxnResources&lt;/tt&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="535362">SERVER-34726</key>
            <summary>Deadlock with locally stashed transaction resources during profiling</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="tess.avitabile@mongodb.com">Tess Avitabile</assignee>
                                    <reporter username="ian.boros@mongodb.com">Ian Boros</reporter>
                        <labels>
                    </labels>
                <created>Fri, 27 Apr 2018 19:17:18 +0000</created>
                <updated>Sun, 29 Oct 2023 22:32:17 +0000</updated>
                            <resolved>Fri, 18 May 2018 21:08:53 +0000</resolved>
                                                    <fixVersion>4.0.0-rc0</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="1895909" author="xgen-internal-githook" created="Fri, 18 May 2018 21:06:24 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;tessavitabile&apos;, &apos;name&apos;: &apos;Tess Avitabile&apos;, &apos;email&apos;: &apos;tess.avitabile@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34726&quot; title=&quot;Deadlock with locally stashed transaction resources during profiling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34726&quot;&gt;&lt;del&gt;SERVER-34726&lt;/del&gt;&lt;/a&gt; Lock acquisitions for profiling in transaction have 0 second timeout&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/6ab1592260c9b21d802aa65a11d268c0a97b11a7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/6ab1592260c9b21d802aa65a11d268c0a97b11a7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1895905" author="tess.avitabile" created="Fri, 18 May 2018 20:58:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alyson.cabral&quot; class=&quot;user-hover&quot; rel=&quot;alyson.cabral&quot;&gt;alyson.cabral&lt;/a&gt;, for this ticket we will make lock acquisition for profiling transaction operations have 0-second timeout, just like all other lock acquisitions in transaction operations. This is because a hang in profiling would prevent the transaction from making progress and could lead to deadlock. If profiling fails, we log a warning but do not abort the transaction. I wanted to make you aware that some transaction operations may not be profiled in 4.0. We could do work in the future to ensure that profiling transaction operations always succeeds (for example, by not doing them in the same thread as the transaction operation).&lt;/p&gt;</comment>
                            <comment id="1893264" author="tess.avitabile" created="Wed, 16 May 2018 19:33:50 +0000"  >&lt;p&gt;No, this would not affect operation logging.&lt;/p&gt;</comment>
                            <comment id="1893259" author="bruce.lucas@10gen.com" created="Wed, 16 May 2018 19:28:41 +0000"  >&lt;p&gt;Does this affect only profiling, and not operation logging? Logging operations in progress on stepdown or shutdown (along with time taken so far) is important, but I think not profling an operation interrupted by shutdown or stepdown is fine - in fact I think arguably more correct as the operation hasn&apos;t completed so you don&apos;t know how long it would actually take if it had.&lt;/p&gt;</comment>
                            <comment id="1893248" author="tess.avitabile" created="Wed, 16 May 2018 19:18:57 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, as part of this work, we are planning to remove the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/373aae274364751eec424a3665c50b53b06f6746/src/mongo/db/introspect.cpp#L122&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt;&lt;/a&gt; from profiling. The &lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt; was there to make sure that we still profile if we exceed &lt;tt&gt;maxTimeMS&lt;/tt&gt;. As a replacement, we will&#160;temporarily suspend the &lt;tt&gt;maxTimeMS&lt;/tt&gt; on the operation, so that we will make sure to profile even if the operation exceeded its &lt;tt&gt;maxTimeMS&lt;/tt&gt;. But removing the &lt;tt&gt;UninterruptibleLockGuard&lt;/tt&gt; would mean we would now fail to profile if a stepdown or shutdown occurred. Does this seem acceptable to you?&lt;/p&gt;</comment>
                            <comment id="1879572" author="spencer" created="Tue, 1 May 2018 20:01:10 +0000"  >&lt;p&gt;&#160;Oh hey, somehow I missed that line!&#160; Looks good then, thanks.&lt;/p&gt;</comment>
                            <comment id="1879456" author="james.wahlin@10gen.com" created="Tue, 1 May 2018 18:49:41 +0000"  >&lt;p&gt;Profiling is enabled in read_concern_snapshot_yielding.js on &lt;a href=&quot;https://github.com/mongodb/mongo/blob/97245f7c13ee8d18e3d82fc87dd44b9fd72b4da3/jstests/noPassthrough/read_concern_snapshot_yielding.js#L40&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this line&lt;/a&gt;. I am not sure if we have a repro outside of the failure in&#160;BF-8848 (&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ian.boros&quot; class=&quot;user-hover&quot; rel=&quot;ian.boros&quot;&gt;ian.boros&lt;/a&gt; - please confirm).&lt;/p&gt;</comment>
                            <comment id="1879443" author="spencer" created="Tue, 1 May 2018 18:44:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ian.boros&quot; class=&quot;user-hover&quot; rel=&quot;ian.boros&quot;&gt;ian.boros&lt;/a&gt;, is there a repro for this?&#160; Did this fail in evergreen?&#160; The linked BF is on read_concern_snapshot_yielding.js, which doesn&apos;t have profiling enabled, so I don&apos;t see how it could trigger this issue?&lt;/p&gt;</comment>
                            <comment id="1876602" author="spencer" created="Fri, 27 Apr 2018 19:55:22 +0000"  >&lt;p&gt;I think another way to fix this would be to make sure that the lock acquisitions done as part of profiling follow has the same 0 second lock acquisition timeout that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33244&quot; title=&quot;Make all lock acquisitions for transactions have 0 second timeout&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33244&quot;&gt;&lt;del&gt;SERVER-33244&lt;/del&gt;&lt;/a&gt; is going to introduce.  Even though the profiling writes are done in a separate storage engine transaction, they are still logically part of the same mongodb transaction, and they prevent the mongodb transaction from making progress, so they should follow the same rules.  Thus if there&apos;s any contention on the locks needed to write to the profile, the transaction should abort immediately.&lt;/p&gt;</comment>
                            <comment id="1876556" author="ian.boros" created="Fri, 27 Apr 2018 19:17:34 +0000"  >&lt;p&gt;If the suggested approach is taken we should be sure not to set the Locker&apos;s thread ID to &quot;unset&quot; when putting the &lt;tt&gt;TxnResources&lt;/tt&gt; back on the session. This will make the LockManager dump output more readable.&lt;/p&gt;

&lt;p&gt;We would also need to add checks &lt;em&gt;after&lt;/em&gt; &lt;tt&gt;profile()&lt;/tt&gt; is called to see if the &lt;tt&gt;TxnResources&lt;/tt&gt; are gone. In this case, we would have to return an error to the user, indicating that the transaction had been aborted.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="495964">SERVER-33244</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="535390">SERVER-34731</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 27 Apr 2018 19:43:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 38 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-33244'>SERVER-33244</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-718</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 38 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>63.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>ian.boros@mongodb.com</customfieldvalue>
            <customfieldvalue>james.wahlin@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htwnfj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9la7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2278">Repl 2018-05-21</customfieldvalue>
    <customfieldvalue id="2296">Repl 2018-06-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htw9nz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>