<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:07:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44914] A shard receiving its first chunk should locally drop any indexes not on the donor</title>
                <link>https://jira.mongodb.org/browse/SERVER-44914</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;From &lt;a href=&quot;https://docs.google.com/document/d/1xBPYqJz8ahrAOG-Rb9XdTeZYTPGvIY_L6YXJ2KaH95M/edit?ts=5dd86765#heading=h.nfk0y78dcoqn&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the design document&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;When a shard moves away its last chunk, it does not drop the collection or any indexes locally, creating an &quot;orphaned&quot; collection. Mongos will only target the shards that own chunks according to its routing table, so an index on an orphaned collection may become inconsistent with the indexes or collection options on chunk owning shards.&lt;/p&gt;

&lt;p&gt;To handle this, when a non-primary shard receives its first chunk for a collection, it will drop the collection if it exists locally before copying the donor&apos;s indexes.&lt;/p&gt;

&lt;p&gt;The primary shard for a collection should always have correct indexes so it must &lt;b&gt;not&lt;/b&gt; drop the collection when it receives its first chunk.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The recipient shard should verify its DatabaseShardingState is up to date before checking if it is the primary shard.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1031544">SERVER-44914</key>
            <summary>A shard receiving its first chunk should locally drop any indexes not on the donor</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="mihai.andrei@mongodb.com">Mihai Andrei</assignee>
                                    <reporter username="jack.mulrow@mongodb.com">Jack Mulrow</reporter>
                        <labels>
                    </labels>
                <created>Mon, 2 Dec 2019 22:44:36 +0000</created>
                <updated>Sun, 29 Oct 2023 22:14:30 +0000</updated>
                            <resolved>Fri, 3 Jan 2020 13:59:55 +0000</resolved>
                                                    <fixVersion>4.3.3</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="2658969" author="xgen-internal-githook" created="Fri, 20 Dec 2019 23:05:45 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Mihai Andrei&apos;, &apos;email&apos;: &apos;mihai.andrei@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44914&quot; title=&quot;A shard receiving its first chunk should locally drop any indexes not on the donor&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44914&quot;&gt;&lt;del&gt;SERVER-44914&lt;/del&gt;&lt;/a&gt; A shard receiving its first chunk should locally drop any indexes not on the donor&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4dc5811a2c427f35ae9dfe5cd110dcdf318bcf9d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4dc5811a2c427f35ae9dfe5cd110dcdf318bcf9d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2650573" author="jack.mulrow" created="Thu, 19 Dec 2019 16:49:00 +0000"  >&lt;p&gt;To keep the primary shard the authoritative source for the collections on a database (unsharded and sharded) we changed this ticket so the recipient will only drop its indexes when receiving its first chunk, instead of dropping the collection.&lt;/p&gt;</comment>
                            <comment id="2601790" author="jack.mulrow" created="Thu, 12 Dec 2019 15:35:32 +0000"  >&lt;p&gt;After further discussion, we&apos;re dropping the requirement that the primary shard always has correct indexes when it has no chunks, so step 1) from my first comment is unnecessary.&lt;/p&gt;</comment>
                            <comment id="2600311" author="jack.mulrow" created="Thu, 12 Dec 2019 00:11:37 +0000"  >&lt;p&gt;Spoke offline with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt; and she pointed out that the recipient shard in a migration &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a1c6a0313164d4a566aaa5f125803adc18772cce/src/mongo/db/s/migration_destination_manager_legacy_commands.cpp#L101&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;already refreshes its metadata before cloning documents&lt;/a&gt;, so we can simplify the implementation from my comment if we move the refresh after &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a1c6a0313164d4a566aaa5f125803adc18772cce/src/mongo/db/s/migration_destination_manager_legacy_commands.cpp#L108-L110&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the recipient registers the migration&lt;/a&gt;. With that change, the recipient should be able to always use its latest CollectionMetadata to check if it owns chunks without needing another refresh.&lt;/p&gt;

&lt;p&gt;She also pointed out that getting the primary shard id from the CatalogCache is unsafe even after a database refresh. Instead we can start storing the primary shard id in the DatabaseShardingState and use that, but since we&apos;re reconsidering the requirement that the primary shard has authoritative indexes, we might be able to throw out the primary check entirely.&lt;/p&gt;</comment>
                            <comment id="2598707" author="jack.mulrow" created="Wed, 11 Dec 2019 14:18:16 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;, can you look over the following proposed implementation?&lt;/p&gt;

&lt;p&gt;The main idea is that because a shard can only be the recipient in one migration at a time, if it refreshes during the receiving logic, it should know definitively if it owns any chunks. I figure adding a refresh to each migration isn&apos;t great though, so I added some optimizations to avoid refreshes in most cases.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Right before a recipient shard &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f515d2ad5494e64c3be80189e7ea6bceaf267421/src/mongo/db/s/migration_destination_manager.cpp#L520&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;clones collection options and indexes from the donor shard&lt;/a&gt;, make it check if it is the primary shard for the migration namespace. If it is the primary, continue the migration as normal, otherwise proceed to step 2.
	&lt;ol&gt;
		&lt;li&gt;Done by first checking if its DatabaseShardingState has a valid version for the migration database, refreshing from the config server through forceDatabaseRefresh() if it doesn&apos;t, and then checking the primaryId from the CachedDatabaseInfo returned by CatalogCache::getDatabase() (called by accessing the CatalogCache through the Grid service context decoration).
		&lt;ol&gt;
			&lt;li&gt;This was based on what we talked about last week, but I&apos;m a little iffy on database version, so let me know if I misunderstood.&lt;/li&gt;
		&lt;/ol&gt;
		&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Make the recipient get the latest filtering metadata for the migration namespace, and only if there is valid metadata (i.e. not boost::none) and it says the recipient owns at least one chunk, continue with the migration as normal, otherwise proceed to step 3.
	&lt;ol&gt;
		&lt;li&gt;The filtering metadata will be retrieved by calling getCurrentMetadataIfKnown() on the CollectionShardingRuntime service context decoration after taking an IS lock on the migration namespace&lt;/li&gt;
		&lt;li&gt;Checking for chunk ownership will be done by either by checking the size of CollectionMetadata::getChunks() on the filtering metadata, or through a new method that calls through into the filtering metadata&apos;s ChunkManager&lt;/li&gt;
		&lt;li&gt;This relies on the assumption that a shard can&apos;t move away its last chunk without either correctly updating its metadata or invalidating it (which is why I think &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44598&quot; title=&quot;Shards do not treat the IGNORED version as &amp;quot;intended sharded&amp;quot;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44598&quot;&gt;&lt;del&gt;SERVER-44598&lt;/del&gt;&lt;/a&gt; shouldn&apos;t be a problem)&lt;/li&gt;
		&lt;li&gt;Note that if the recipient&apos;s metadata says it owns no chunks, it still needs to refresh its metadata to handle the case where it received a chunk but hasn&apos;t refreshed yet&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Make the recipient refresh its metadata for the migration namespace from the config server
	&lt;ol&gt;
		&lt;li&gt;Done by calling forceShardFilteringMetadataRefresh() after dropping the IS lock taken earlier&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Check if the recipient owns at least one chunk using the refreshed metadata. If the shard still does not own a chunk, make the recipient drop the migration namespace locally before continuing with the migration&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Technically it&apos;s possible after refreshing in step 3 the recipient moves away its last chunk and incorrectly decides it doesn&apos;t need to drop the collection in step 4, but I don&apos;t think this is a problem because mongos shard versions index operations. If there&apos;s a concurrent index operation it must target the shard donating a chunk to this recipient, and the operation will either execute on that donor before this migration reaches its critical section, which will abort the migration, or it executes after and the command will fail with StaleConfig and be retried, and the retry should correctly target the recipient.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1031573">SERVER-44918</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="2433338">SERVER-80703</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Dec 2019 16:59:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 7 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1361</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 7 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jack.mulrow@mongodb.com</customfieldvalue>
            <customfieldvalue>mihai.andrei@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw79rj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvvhsf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3473">Sharding 2019-12-16</customfieldvalue>
    <customfieldvalue id="3474">Sharding 2019-12-30</customfieldvalue>
    <customfieldvalue id="3475">Sharding 2020-01-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw6w0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>