<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:45:35 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-81133] Speedup logic to persist routing table cache</title>
                <link>https://jira.mongodb.org/browse/SERVER-81133</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Each shard maintain a persisted version of the collection routing table cache in &lt;tt&gt;config.cache.chunks.&amp;lt;nss&amp;gt;&lt;/tt&gt;. The function used to update this collection is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/669f87e05810e70c9586a5ff46d44aeb550992ad/src/mongo/db/s/shard_metadata_util.cpp#L361&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;updateShardChunks&lt;/a&gt; that is performing sequentially a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/669f87e05810e70c9586a5ff46d44aeb550992ad/src/mongo/db/s/shard_metadata_util.cpp#L406-L418&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;delete operation&lt;/a&gt; followed by an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/669f87e05810e70c9586a5ff46d44aeb550992ad/src/mongo/db/s/shard_metadata_util.cpp#L420-L428&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;insert operation&lt;/a&gt; for every new chunk.&lt;/p&gt;

&lt;p&gt;When the number of new chunks to persist is high, this logic could become extremely slow. For instance, with a routing table of 10 millions chunks, it could take up to 45 min to persist the entire routing table.&lt;/p&gt;

&lt;p&gt;This is problematic because until all the updated chunks get persisted, they will slow down considerably all the concurrent incremental routing table refreshes. For instance, an incremental refresh for a routing table of 10 millions chunks usually takes on the order of 10 milliseconds, but if the routing table needs still to be persisted to disk this incremental refresh can take up to ~5 secs.&lt;/p&gt;

&lt;p&gt;The goal of this ticket is to speedup the logic that persist new chunks in &lt;tt&gt;config.cache.chunks.&amp;lt;nss&amp;gt;&lt;/tt&gt;. One potential solution could be to perform a batch of deletions followed by a batch of insertion.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2444457">SERVER-81133</key>
            <summary>Speedup logic to persist routing table cache</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="tommaso.tocci@mongodb.com">Tommaso Tocci</assignee>
                                    <reporter username="tommaso.tocci@mongodb.com">Tommaso Tocci</reporter>
                        <labels>
                    </labels>
                <created>Mon, 18 Sep 2023 07:21:36 +0000</created>
                <updated>Tue, 30 Jan 2024 22:35:39 +0000</updated>
                            <resolved>Thu, 21 Sep 2023 07:02:40 +0000</resolved>
                                    <version>7.0.1</version>
                    <version>4.4.24</version>
                    <version>6.0.10</version>
                    <version>5.0.21</version>
                    <version>7.1.0-rc2</version>
                                    <fixVersion>7.1.1</fixVersion>
                    <fixVersion>7.2.0-rc0</fixVersion>
                    <fixVersion>7.0.4</fixVersion>
                    <fixVersion>6.0.13</fixVersion>
                    <fixVersion>5.0.25</fixVersion>
                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="6010380" author="xgen-internal-githook" created="Sun, 14 Jan 2024 12:30:23 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81133&quot; title=&quot;Speedup logic to persist routing table cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81133&quot;&gt;&lt;del&gt;SERVER-81133&lt;/del&gt;&lt;/a&gt; Speedup logic to persist routing table cache&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 5a8d10b236e54bd03cae15d453cdc90c466d6168)&lt;br/&gt;
(cherry picked from commit 7db3a426f1af829be41bd3d8033049f2ca8deea3)&lt;br/&gt;
(cherry picked from commit ae850b2403e0e7b41062d626d3d0158c96f81cfe)&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: 8266e4685572c669de36c2139af8949ff7bb9c90&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/7060b169e891499168909f54f246e9a8e9a59c48&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/7060b169e891499168909f54f246e9a8e9a59c48&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5961485" author="xgen-internal-githook" created="Tue, 19 Dec 2023 22:16:42 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81133&quot; title=&quot;Speedup logic to persist routing table cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81133&quot;&gt;&lt;del&gt;SERVER-81133&lt;/del&gt;&lt;/a&gt; Speedup logic to persist routing table cache&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 5a8d10b236e54bd03cae15d453cdc90c466d6168)&lt;br/&gt;
(cherry picked from commit 7db3a426f1af829be41bd3d8033049f2ca8deea3)&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: ae850b2403e0e7b41062d626d3d0158c96f81cfe&lt;br/&gt;
Branch: v6.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8bf71fc4d088b59d45f927f2936d8c60ce134cb0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8bf71fc4d088b59d45f927f2936d8c60ce134cb0&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5861584" author="xgen-internal-githook" created="Tue, 7 Nov 2023 18:49:01 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81133&quot; title=&quot;Speedup logic to persist routing table cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81133&quot;&gt;&lt;del&gt;SERVER-81133&lt;/del&gt;&lt;/a&gt; Speedup logic to persist routing table cache&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 5a8d10b236e54bd03cae15d453cdc90c466d6168)&lt;br/&gt;
Branch: v7.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9f835bf70eb528583a5f9542144cb4f083e8ed97&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9f835bf70eb528583a5f9542144cb4f083e8ed97&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5856326" author="xgen-internal-githook" created="Mon, 6 Nov 2023 11:14:21 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81133&quot; title=&quot;Speedup logic to persist routing table cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81133&quot;&gt;&lt;del&gt;SERVER-81133&lt;/del&gt;&lt;/a&gt; Speedup logic to persist routing table cache&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 5a8d10b236e54bd03cae15d453cdc90c466d6168)&lt;br/&gt;
Branch: v7.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e685ee8f9abe164ec0baf5fad1cc3c09103d6e15&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/e685ee8f9abe164ec0baf5fad1cc3c09103d6e15&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5719242" author="xgen-internal-githook" created="Wed, 20 Sep 2023 19:31:06 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-81133&quot; title=&quot;Speedup logic to persist routing table cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-81133&quot;&gt;&lt;del&gt;SERVER-81133&lt;/del&gt;&lt;/a&gt; Speedup logic to persist routing table cache&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/5a8d10b236e54bd03cae15d453cdc90c466d6168&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/5a8d10b236e54bd03cae15d453cdc90c466d6168&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5718972" author="tommaso.tocci" created="Wed, 20 Sep 2023 18:28:35 +0000"  >&lt;p&gt;I&apos;ve performed some testing on the X64 workstation and these are the results:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Without batching:
	&lt;ul&gt;
		&lt;li&gt;&lt;tt&gt;1M&lt;/tt&gt; chunks: ~195 seconds&lt;/li&gt;
		&lt;li&gt;&lt;tt&gt;2&lt;/tt&gt; chunks: between 1 and 2 milliseconds&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;With batching (&lt;tt&gt;1000&lt;/tt&gt;):
	&lt;ul&gt;
		&lt;li&gt;&lt;tt&gt;1M&lt;/tt&gt; chunks: ~48 seconds&lt;/li&gt;
		&lt;li&gt;&lt;tt&gt;2&lt;/tt&gt; chunks: less than 1 millisecond&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="2447691">DOCS-16394</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="2440698">SERVER-81016</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25133"><![CDATA[Sharding EMEA]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="26243"><![CDATA[v7.1]]></customfieldvalue>
    <customfieldvalue key="25578"><![CDATA[v7.0]]></customfieldvalue>
    <customfieldvalue key="23470"><![CDATA[v6.0]]></customfieldvalue>
    <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001yJQpyQAG]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 20 Sep 2023 10:07:47 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Introduced a new parameter `persistedChunkCacheUpdateMaxBatchSize` on mongod to control the maximum batch size used for updating persisted chunk cache on shards.&lt;br/&gt;
&lt;br/&gt;
The parameter is defaulted to 1000 and can be changed both at runtime and startup.</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>35.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2qxfb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i28vwg:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7719">Sharding EMEA 2023-10-02</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16943"><![CDATA[Cloud]]></customfieldvalue>
    <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    <customfieldvalue key="16946"><![CDATA[Triage and Release]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2qjkn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>