<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:28:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31732] Recipient shards of migrations hold DBLock in X mode while creating indexes</title>
                <link>https://jira.mongodb.org/browse/SERVER-31732</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Recipient shards take the DBLock twice, during the first two steps in the MigrationDestinationManager:&lt;/p&gt;

&lt;p&gt;Step 0 (&quot;copy system.namespaces entry if collection doesn&apos;t already exist&quot;): &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/s/migration_destination_manager.cpp#L514&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;takes the DBLock&lt;/a&gt; in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/db_raii.cpp?#L341-L347&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mode IX if it has the collection, else in mode X&lt;/a&gt;  so the collection can be created&lt;/p&gt;

&lt;p&gt;Step 1 (&quot;copy indexes&quot;): &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/s/migration_destination_manager.cpp#L595&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;takes the DBLock in mode X&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It may be unnecessarily heavy-handed to take the DBLock in mode X in Step 1 (although, createIndexes command itself does &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/commands/create_indexes.cpp#L254&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;take the DBLock in mode X&lt;/a&gt; in case it needs to create the collection, but &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/commands/create_indexes.cpp#L328-L332&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;only relocks it with mode IX if the index was asked to be built in the background&lt;/a&gt;. createIndexes does &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/commands/create_indexes.cpp#L347-L354&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;relock the DBLock in mode X regardless to complete the index build&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Note that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31485&quot; title=&quot;Race between move chunks and dropIndex may lead to IndexNotFound error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31485&quot;&gt;&lt;del&gt;SERVER-31485&lt;/del&gt;&lt;/a&gt; will make the MigrationDestinationManager hold the DBLock across both steps.&lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kevin.pulo&quot; class=&quot;user-hover&quot; rel=&quot;kevin.pulo&quot;&gt;kevin.pulo&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="451466">SERVER-31732</key>
            <summary>Recipient shards of migrations hold DBLock in X mode while creating indexes</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="backlog-server-sharding">[DO NOT USE] Backlog - Sharding Team</assignee>
                                    <reporter username="esha.maharishi@mongodb.com">Esha Maharishi</reporter>
                        <labels>
                    </labels>
                <created>Thu, 26 Oct 2017 16:05:59 +0000</created>
                <updated>Fri, 27 Oct 2023 13:54:08 +0000</updated>
                            <resolved>Mon, 6 Nov 2017 19:33:52 +0000</resolved>
                                    <version>3.6.0-rc1</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1717899" author="kaloian.manassiev" created="Mon, 6 Nov 2017 19:33:52 +0000"  >&lt;p&gt;Creating indexes requires holding an X-lock on the database, because the index catalog is per-database. This is something, which needs to be fixed at the storage engine level.&lt;/p&gt;</comment>
                            <comment id="1713539" author="esha.maharishi@10gen.com" created="Tue, 31 Oct 2017 15:12:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt; yes, I realized that when started testing - it looks like it&apos;s more of an &quot;auto-healing&quot; behavior, where missing indexes are created on chunk migration. Thanks for mentioning it &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1713531" author="renctan" created="Tue, 31 Oct 2017 15:07:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt; I think the removeExistingIndexes doesn&apos;t actually remove the indexes in the catalog, but filters out the indexes in the argument that has been already created.&lt;/p&gt;</comment>
                            <comment id="1710413" author="milkie" created="Thu, 26 Oct 2017 18:41:07 +0000"  >&lt;p&gt;Removing all existing indexes sounds like a bandaid from back in the day that we could improve.&lt;br/&gt;
However, failing migration when indexes don&apos;t match might be too ambitious before we can make more headway on Collection Lifecycle.  I believe there are just too many existing races in index management today that can easily result in indexes getting out of sync across shards (but I could be wrong!).&lt;/p&gt;</comment>
                            <comment id="1710406" author="esha.maharishi@10gen.com" created="Thu, 26 Oct 2017 18:35:17 +0000"  >&lt;p&gt;You&apos;re right, the index should already be there &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; It seems like the recipient shard actually &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.0-rc1/src/mongo/db/s/migration_destination_manager.cpp#L611&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;removes all existing indexes&lt;/a&gt; and re-builds them each time it accepts a chunk...&lt;/p&gt;

&lt;p&gt;I&apos;m hoping to change this so that the recipient shard only builds the indexes if it&apos;s accepting its first chunk for the collection. If it&apos;s any later chunk, it will fail the migration if its indexes differ from what&apos;s on the donor.&lt;/p&gt;</comment>
                            <comment id="1710395" author="milkie" created="Thu, 26 Oct 2017 18:28:47 +0000"  >&lt;p&gt;My first thought is: why is there any data in the collection at all when these indexes are built?  Shouldn&apos;t the collection be empty, and the index build time very short?  &lt;br/&gt;
If there is data in the collection, shouldn&apos;t the index already be there; why would a chunk migration trigger it?&lt;/p&gt;</comment>
                            <comment id="1710392" author="milkie" created="Thu, 26 Oct 2017 18:27:05 +0000"  >&lt;p&gt;Yes, migration manager is using the MultiIndexBlock builder, so it needs to follow the same locking rules that create_indexes.cpp uses.  You could switch it to use the background index locking scheme, but that would have repercussions (it would make chunk migration even slower, and would result in larger, less efficient initial B-trees for those indexes).&lt;/p&gt;</comment>
                            <comment id="1710381" author="esha.maharishi@10gen.com" created="Thu, 26 Oct 2017 18:20:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, got it. So we should always hold a DB X lock while creating indexes for the reasons you mentioned, including in the migration code?&lt;/p&gt;</comment>
                            <comment id="1710379" author="milkie" created="Thu, 26 Oct 2017 18:18:44 +0000"  >&lt;p&gt;Minor technicality: The createIndexes command takes a DB X lock because it needs to update the catalog metadata for the database to add the new index entry, regardless of whether the collection exists already.  It also reacquires the DB X lock at the end of a background index build, as it needs to update the catalog metadata again to mark the index as &quot;ready&quot;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="443211">SERVER-31485</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25141"><![CDATA[Sharding]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 26 Oct 2017 18:18:44 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 14 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-629</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 14 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-sharding</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htisyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htalrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htif2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>