<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:33:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-33356] Ensure shards&apos; persisted collection cache picks up collection UUIDs after setFCV=4.0</title>
                <link>https://jira.mongodb.org/browse/SERVER-33356</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;FCV 3.4-&amp;gt;3.6 has no schema upgrade process for config.cache.collections.&lt;/p&gt;

&lt;p&gt;config.cache.collections was added to shards as part of the safe secondary reads project, which didn&apos;t require an upgrade process, and safely started persisting routing metadata as soon as the v3.6 binary cluster started up. However, UUIDs were added to config.cache.collections to address replication change streams needs. I believe that a config config.collections schema upgrade process was added, but no one thought to add an upgrade process for shards.&lt;/p&gt;

&lt;p&gt;So, ShardServerCatalogCacheLoader will happily persist config.cache.collections from the start. Then setFCV(3.6) happens, and config.cache.collections won&apos;t get the UUID until the shard refreshes NEW chunks for that collection.&lt;/p&gt;

&lt;p&gt;The ShardServerCatalogCacheLoader won&apos;t schedule a persisted update unless new chunks are received from the config server, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/6bd287f8805027ae88f2d22df0bb1af3b6d6753b/src/mongo/db/s/shard_server_catalog_cache_loader.cpp#L508-L518&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;per this bit of code&lt;/a&gt;. That was added to improve secondary availability because every primary persistence task sets flags that block metadata reads, and might even prompt forcing the secondary to refresh if we got around to that (I forget).&lt;/p&gt;

&lt;p&gt;Consider the in-memory UUID values in the upgrade process. ChunkManager is probably fine because it gets recreated/updated every CatalogCache refresh. Unsure if we have any other in-memory UUIDs fields that must get set on upgrade.&lt;/p&gt;</description>
                <environment></environment>
        <key id="498723">SERVER-33356</key>
            <summary>Ensure shards&apos; persisted collection cache picks up collection UUIDs after setFCV=4.0</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="esha.maharishi@mongodb.com">Esha Maharishi</assignee>
                                    <reporter username="dianna.hohensee@mongodb.com">Dianna Hohensee</reporter>
                        <labels>
                    </labels>
                <created>Thu, 15 Feb 2018 20:01:03 +0000</created>
                <updated>Sun, 29 Oct 2023 22:34:38 +0000</updated>
                            <resolved>Mon, 16 Apr 2018 18:02:14 +0000</resolved>
                                                    <fixVersion>3.7.4</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="3601075" author="xgen-internal-githook" created="Fri, 5 Feb 2021 10:17:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jordi Serra Torrens&apos;, &apos;email&apos;: &apos;jordi.serra-torrens@mongodb.com&apos;, &apos;username&apos;: &apos;jordist&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53274&quot; title=&quot;The force-write of the `history` field to all chunks on FCV upgrade from 3.6 to 4.0 is unnecessary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53274&quot;&gt;&lt;del&gt;SERVER-53274&lt;/del&gt;&lt;/a&gt; Avoid bumping all chunk versions after writing &apos;history&apos; field on FCV upgrade to 4.0.&lt;/p&gt;

&lt;p&gt;Instead, bump only one chunk per shard to satisfy the reqirements&lt;br/&gt;
imposed by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33356&quot; title=&quot;Ensure shards&amp;#39; persisted collection cache picks up collection UUIDs after setFCV=4.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33356&quot;&gt;&lt;del&gt;SERVER-33356&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/52d6f11c459b8d3666379431a6accf7fef4e852f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/52d6f11c459b8d3666379431a6accf7fef4e852f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2007058" author="esha.maharishi@10gen.com" created="Tue, 18 Sep 2018 19:53:38 +0000"  >&lt;p&gt;Note, this patch committed a test that demonstrated that shards would pick up collection UUIDs &lt;b&gt;the next time the shards refreshed&lt;/b&gt;&#160;after a setFCV=4.0.&lt;/p&gt;

&lt;p&gt;However, change streams (which, as of 4.0, are the only consumer of UUIDs in the shards&apos; routing table caches) on sharded collections are unversioned, so they do not cause the shard to refresh.&lt;/p&gt;

&lt;p&gt;-------&lt;/p&gt;

&lt;p&gt;Further, this &lt;a href=&quot;https://github.com/mongodb/mongo/commit/a58a24439b972bb8af00caf0cf6c0a8696a5899c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;later commit&lt;/a&gt;&#160;updated the test to use a findOne, rather than a forced refresh, against the shard to make the shard refresh (since findOne is versioned).&lt;/p&gt;

&lt;p&gt;However,&lt;/p&gt;

&lt;p&gt;1) By the time the later commit went in, setFCV was &lt;a href=&quot;https://github.com/mongodb/mongo/blob/c02574298a711b6de8a3d89cedcfe98040a6f55b/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L206&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;updating the chunks collection on the config server&lt;/a&gt;&#160;with &lt;a href=&quot;https://github.com/mongodb/mongo/blob/c02574298a711b6de8a3d89cedcfe98040a6f55b/src/mongo/db/s/config/sharding_catalog_manager_chunk_operations.cpp#L822-L825&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bumped chunk versions&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;2) The router was refreshing and picking up the bumped chunk versions &lt;b&gt;before&lt;/b&gt; sending the versioned request to the shard. This is because the router was invalidating its in-memory cache on the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a58a24439b972bb8af00caf0cf6c0a8696a5899c/jstests/sharding/shard_collection_cache_upgrade_downgrade.js#L53-L54&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shardCollection calls&lt;/a&gt;, so the router would pick up the bumped chunk versions on the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a58a24439b972bb8af00caf0cf6c0a8696a5899c/jstests/sharding/shard_collection_cache_upgrade_downgrade.js#L111-L112&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;findOne calls&lt;/a&gt;&#160;&lt;b&gt;before&lt;/b&gt;&#160;sending the findOne calls to the shard. Since the router&apos;s requests would have the bumped chunk versions, of course the shard would detect a version mismatch and refresh.&lt;/p&gt;

&lt;p&gt;The test is not demonstrating that a&#160;&lt;b&gt;stale&lt;/b&gt; router would cause the shard to refresh.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="1865452" author="xgen-internal-githook" created="Mon, 16 Apr 2018 18:01:52 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Esha Maharishi&apos;, &apos;email&apos;: &apos;esha.maharishi@mongodb.com&apos;, &apos;username&apos;: &apos;EshaMaharishi&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33356&quot; title=&quot;Ensure shards&amp;#39; persisted collection cache picks up collection UUIDs after setFCV=4.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33356&quot;&gt;&lt;del&gt;SERVER-33356&lt;/del&gt;&lt;/a&gt; Ensure shards&apos; persisted collection cache picks up collection UUIDs after setFCV=4.0&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c02574298a711b6de8a3d89cedcfe98040a6f55b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c02574298a711b6de8a3d89cedcfe98040a6f55b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1811692" author="kaloian.manassiev" created="Wed, 21 Feb 2018 16:32:10 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt;, as part of investigation for this bug it was discovered that there are possible situations in 3.6 where the shard filtering metadata collections will be missing collection UUIDs. We will fix it for FCV 4.0, but backporting it for existing 3.6 deployments is problematic since we don&apos;t have the FCV upgrade action to hook it into.&lt;/p&gt;

&lt;p&gt;I believe the only user of the collection UUIDs in 3.6 is change streams. Do you know what problems can arise if the UUIDs are missing? I would like to gauge how bad it is so we don&apos;t have to figure out a 3.6 solution.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="508776">SERVER-33783</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="500250">SERVER-33401</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 21 Feb 2018 16:32:10 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-33783'>SERVER-33783</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htqj2n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8zyv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2216">Sharding 2018-04-09</customfieldvalue>
    <customfieldvalue id="2217">Sharding 2018-04-23</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htq587:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>