<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:32:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53846] Clarify locking requirements in createCollectionForApplyOps</title>
                <link>https://jira.mongodb.org/browse/SERVER-53846</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I was looking at how we make createCollection idempotent for oplog application. In the case there is a conflicting collection with the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0edda21e200d7d7186f236e54bb331c402cdeb1a/src/mongo/db/catalog/create_collection.cpp#L480-L481&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;same namespace&lt;/a&gt;, we would&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0edda21e200d7d7186f236e54bb331c402cdeb1a/src/mongo/db/catalog/create_collection.cpp#L513&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rename&lt;/a&gt;&#160;the existing collection (&lt;tt&gt;newCollName&lt;/tt&gt;) to a temporary collection (&lt;tt&gt;tmpName&lt;/tt&gt;). But&#160;&lt;tt&gt;renameCollection&lt;/tt&gt;&#160;requires a&#160;&lt;tt&gt;MODE_X&lt;/tt&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0edda21e200d7d7186f236e54bb331c402cdeb1a/src/mongo/db/catalog/database_impl.cpp#L529-L530&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&#160;lock on both the&#160;&lt;/a&gt;&lt;tt&gt;fromNss&lt;/tt&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0edda21e200d7d7186f236e54bb331c402cdeb1a/src/mongo/db/catalog/database_impl.cpp#L529-L530&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&#160;and the&#160;&lt;/a&gt;&lt;tt&gt;toNss&lt;/tt&gt;. But based on what I saw in&#160;&lt;tt&gt;createCollectionForApplyOps&lt;/tt&gt;, I don&apos;t see how we could hold a&#160;&lt;tt&gt;MODE_X&lt;/tt&gt;&#160;lock on&#160;&lt;tt&gt;newCollName&lt;/tt&gt;&#160;.&#160;Same for this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/14ad73ec6ab5eb8026b0577e22ba357e1b7b3bf8/src/mongo/db/catalog/create_collection.cpp#L562&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rename&lt;/a&gt;. I don&apos;t see necessary locks being acquired in this function.&lt;/p&gt;

&lt;p&gt;I initially suspected this was a bug. But actually we won&apos;t hit the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/14ad73ec6ab5eb8026b0577e22ba357e1b7b3bf8/src/mongo/db/catalog/database_impl.cpp#L529-L530&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;locking invariants in &lt;tt&gt;renameCollection&lt;/tt&gt;&lt;/a&gt; because because&#160;&lt;tt&gt;isCollectionLockedForMode&lt;/tt&gt;&#160;always returns true when&#160;&lt;tt&gt;!shouldConflictWithSecondaryBatchApplication()&lt;/tt&gt;. And currently, the oplogApplier always &lt;a href=&quot;https://github.com/mongodb/mongo/blob/6364c95692fbf3cd48c5ba13772eede90b61fe86/src/mongo/db/repl/oplog_applier_impl.cpp#L782&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;sets shouldConflictWithSecondaryBatchApplication to false&lt;/a&gt;. If this is the assumption we need for calling &lt;tt&gt;createCollectionForApplyOps&lt;/tt&gt;, we should add an invariant.&lt;/p&gt;

&lt;p&gt;The &lt;a href=&quot;https://github.com/mongodb/mongo/blob/14ad73ec6ab5eb8026b0577e22ba357e1b7b3bf8/src/mongo/db/catalog/database_impl.cpp#L529-L530&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;locking invariants in &lt;tt&gt;renameCollection&lt;/tt&gt;&lt;/a&gt; were hit during tenant migrations. Tenant migration is similar to initial sync but it applies oplog entries on the primary node. So we don&apos;t set shouldConflictWithSecondaryBatchApplication to false like we do for normal replication. To work around the invariants mentioned above and unblock tenant migration, we could make tenant migration set shouldConflictWithSecondaryBatchApplication to false. But I think &lt;tt&gt;createCollectionForApplyOps&lt;/tt&gt; should probably acquire the necessary locks on its own. So this is a ticket for that investigation and I am not sure if this would be a bug for applyOps (maybe not because applyOps takes global X lock for commands?).&lt;/p&gt;</description>
                <environment></environment>
        <key id="1592660">SERVER-53846</key>
            <summary>Clarify locking requirements in createCollectionForApplyOps</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="gregory.noma@mongodb.com">Gregory Noma</assignee>
                                    <reporter username="lingzhi.deng@mongodb.com">Lingzhi Deng</reporter>
                        <labels>
                    </labels>
                <created>Fri, 15 Jan 2021 22:34:57 +0000</created>
                <updated>Sun, 29 Oct 2023 21:58:43 +0000</updated>
                            <resolved>Wed, 26 Jul 2023 02:20:50 +0000</resolved>
                                                    <fixVersion>7.1.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="5591559" author="xgen-internal-githook" created="Wed, 26 Jul 2023 02:18:09 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Noma&apos;, &apos;email&apos;: &apos;gregory.noma@gmail.com&apos;, &apos;username&apos;: &apos;gregorynoma&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53846&quot; title=&quot;Clarify locking requirements in createCollectionForApplyOps&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53846&quot;&gt;&lt;del&gt;SERVER-53846&lt;/del&gt;&lt;/a&gt; Take collection locks when renaming for applying `create`&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3cec0462cd36c3afd0da14b9986c4458b02f209b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3cec0462cd36c3afd0da14b9986c4458b02f209b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3605397" author="lingzhi.deng" created="Mon, 8 Feb 2021 20:33:39 +0000"  >&lt;p&gt;Just one more thing, we used to acquire proper locks in the function until&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42478&quot; title=&quot;Remove DB MODE_X locks from oplog application of create/createIndexes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42478&quot;&gt;&lt;del&gt;SERVER-42478&lt;/del&gt;&lt;/a&gt;. So I still hope we can sort this out at some point, maybe by 5.0.&lt;/p&gt;</comment>
                            <comment id="3605354" author="dianna.hohensee" created="Mon, 8 Feb 2021 20:19:34 +0000"  >&lt;p&gt;Yes, I definitely agree that the behavior is concerning, particularly the ease with which issues could arise from not realizing new code is unintentionally running without locks thanks to the bypass.&lt;/p&gt;

&lt;p&gt;I&apos;m going to close this investigation as complete. This will be kept in mind for the future. Thanks for looking into it!&lt;/p&gt;</comment>
                            <comment id="3602728" author="lingzhi.deng" created="Fri, 5 Feb 2021 22:32:12 +0000"  >&lt;p&gt;For the time being, I set &lt;tt&gt;shouldConflictWithSecondaryBatchApplication&lt;/tt&gt; to false for tenant migration to work around this so it is not too urgent at the moment. But this was very confusing and I am afraid there might be other code path that could hit this. But I am ok with deferring this to later when we have the bandwidth.&lt;/p&gt;</comment>
                            <comment id="3602617" author="lingzhi.deng" created="Fri, 5 Feb 2021 21:38:04 +0000"  >&lt;p&gt;What do you mean by &quot;main applier thread runs X lock operations serially&quot;?&lt;br/&gt;
In tenant migration, we assume no concurrent accesses to the tenant&apos;s collection while the tenant oplog applier is applying writes. And this assumption is guaranteed by the Atlas proxy as part of the migration protocol. I think ideally &lt;tt&gt;createCollectionForApplyOps&lt;/tt&gt; should be self-contained and acquire whatever locks it needs.&lt;/p&gt;</comment>
                            <comment id="3602131" author="dianna.hohensee" created="Fri, 5 Feb 2021 17:54:01 +0000"  >&lt;p&gt;Apparently, for secondary oplog application, the writer threads expect the main applier thread to hold locks for them. The main applier thread runs X lock operations serially, so no other writers are active in the system. Therefore, &lt;tt&gt;isCollectionLockedForMode&lt;/tt&gt;&#160;returns &lt;tt&gt;true&lt;/tt&gt;&#160;if &lt;tt&gt;shouldConflictWithSecondaryBatchApplication==false&lt;/tt&gt;&#160;as a signal that is what is happening.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lingzhi.deng&quot; class=&quot;user-hover&quot; rel=&quot;lingzhi.deng&quot;&gt;lingzhi.deng&lt;/a&gt;, does tenant migration put the server into a similar mode such that concurrent writes are not a concern? Otherwise, not holding the appropriate locks on some thread, sounds like it could bee an problem.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1590739">SERVER-53817</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="878699">SERVER-42478</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1564264">SERVER-53312</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 5 Feb 2021 17:54:01 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        28 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2170</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            28 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>gregory.noma@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hypyjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1yd8u:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7182">Execution NAMR Team 2023-08-07</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hypksn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>