<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:29:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-75285] Deadlock between ShardsvrCheckMetadataConsistencyParticipantCommand, prepared transactions, and stepdown</title>
                <link>https://jira.mongodb.org/browse/SERVER-75285</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;ShardsvrCheckMetadataConsistencyParticipantCommand currently &lt;a href=&quot;https://github.com/mongodb/mongo/blob/2ff107f9476639d6d139f10228217de017629bd3/src/mongo/db/s/shardsvr_check_metadata_consistency_participant_command.cpp#L90&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;takes a DB lock in &lt;del&gt;S mode&lt;/del&gt; IS mode&lt;/a&gt; without exempting taking the RSTL. This means that it will not be killed on stepdown (since it didn&apos;t &lt;a href=&quot;https://github.com/mongodb/mongo/blob/56d9c847ef0b81902c80c3f8aa4c921049f02a43/src/mongo/db/repl/replication_coordinator_impl.cpp#L2588&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;take the global lock in a mode that conflicts with writes&lt;/a&gt;). &lt;/p&gt;

&lt;p&gt;(Edit: at the time that this deadlock was found, the command &lt;a href=&quot;https://github.com/mongodb/mongo/blob/85492fee5ab876a538f1e3915d26eb9b28ffff2e/src/mongo/db/s/shardsvr_check_metadata_consistency_participant_command.cpp#L92&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;took the DB lock in S mode&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;This can then cause a deadlock with prepared transactions if the transaction is holding the DB lock that checkMetadataConsistency is looking to acquire, but committing the transaction is blocked on a stepdown (as in the node isn&apos;t able to replicate the commitTransaction command until it completes stepping down).&lt;/p&gt;

&lt;p&gt;The order of events is:&lt;br/&gt;
1. Prepare a transaction that holds the DB lock in IX for some db that checkMetadataConsistency might need to take a DB lock for&lt;br/&gt;
2. ShardsvrCheckMetadataConsistencyParticipantCommand tries to take the DB lock for the db mentioned above, ends up holding the RSTL in IX mode while it waits&lt;br/&gt;
3. Node tries to step down before it receives the commitTransaction command &lt;/p&gt;

&lt;p&gt;A targeted way to fix this would be to manually ensure that checkMetadataConsistency is killed by the stepdown thread or make sure it does not hold the RSTL. &lt;/p&gt;</description>
                <environment></environment>
        <key id="2297785">SERVER-75285</key>
            <summary>Deadlock between ShardsvrCheckMetadataConsistencyParticipantCommand, prepared transactions, and stepdown</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="tommaso.tocci@mongodb.com">Tommaso Tocci</assignee>
                                    <reporter username="samy.lanka@mongodb.com">Samyukta Lanka</reporter>
                        <labels>
                    </labels>
                <created>Fri, 24 Mar 2023 20:08:27 +0000</created>
                <updated>Fri, 27 Oct 2023 20:44:51 +0000</updated>
                            <resolved>Mon, 27 Mar 2023 13:23:59 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5302462" author="samy.lanka" created="Mon, 27 Mar 2023 13:38:21 +0000"  >&lt;blockquote&gt;&lt;p&gt;from your explanation it seems like read operations must always be lock-free - am I understanding it correctly?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I think an amendment based on Jordi&apos;s point is that reads that take DB S mode locks should instead be lock free or we do &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-75288&quot; title=&quot;Investigate whether the stepdown killop thread should kill operations that hold the RSTL&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-75288&quot;&gt;SERVER-75288&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="5302412" author="samy.lanka" created="Mon, 27 Mar 2023 13:23:22 +0000"  >&lt;p&gt;That&apos;s a great point, I think &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jordi.serra-torrens%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;jordi.serra-torrens@mongodb.com&quot;&gt;jordi.serra-torrens@mongodb.com&lt;/a&gt; is correct that this can&apos;t happen anymore because the IS lock won&apos;t conflict with prepared transactions. &lt;/p&gt;</comment>
                            <comment id="5301992" author="JIRAUSER1257318" created="Mon, 27 Mar 2023 10:18:47 +0000"  >&lt;p&gt;I&apos;d like to point out that on BF-28038, ShardsvrCheckMetadataConsistencyParticipantCommand &lt;a href=&quot;https://jira.mongodb.org/browse/BF-28038?focusedCommentId=5299466&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-5299466&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;was trying to acquire the DB lock in MODE_S&lt;/a&gt; (rather than IS). The change from S to IS happened as part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-74667&quot; title=&quot;Use lock-free read approch for checkMetadataConsistency command&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-74667&quot;&gt;&lt;del&gt;SERVER-74667&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I think that&apos;s important, because I wouldn&apos;t expect ShardsvrCheckMetadataConsistencyParticipantCommand&apos;s MODE_IS acquisition to be blocked due to the prepared txn (MODE_IX). MODE_S however, would have blocked.&lt;/p&gt;</comment>
                            <comment id="5301965" author="kaloian.manassiev" created="Mon, 27 Mar 2023 10:02:07 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=samy.lanka%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;samy.lanka@mongodb.com&quot;&gt;samy.lanka@mongodb.com&lt;/a&gt;, from your explanation it seems like read operations must always be lock-free - am I understanding it correctly?&lt;/p&gt;

&lt;p&gt;My reading of it is that read operations shouldn&apos;t be holding the RSTL lock while waiting for IS locks further down the hierarchy. But that would only be possible if we had some snapshotting mechanism to ensure the read will access a consistent state, i.e. what is present in lock-free reads.&lt;/p&gt;

&lt;p&gt;Do we have an example of a read operation which must run with locks held?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2297842">SERVER-75288</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2234556">SERVER-72895</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2282951">SERVER-74667</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25133"><![CDATA[Sharding EMEA]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 27 Mar 2023 10:02:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        45 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3020</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            45 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>135.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jordi.serra-torrens@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>samy.lanka@mongodb.com</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i21t27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1k5uo:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i21f7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>