<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:58:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-85590] Mongos startup time increase linearly with the amount of databases in the cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-85590</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;On startup, mongos runs 1 query per database to the `config.collections` collection.&lt;/p&gt;

&lt;p&gt;On cluster with large amount of databases, it will have an important impact on the process startup time.&lt;/p&gt;

&lt;p&gt;Moreover, as mongos uses `ReadConcern: majority` to communicate with config replica, the startup time can also vary depending to the network latency between config members and the mongos.&lt;/p&gt;

&lt;p&gt;Here are the kind of requests sent by mongos to config RS on startup:&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-22T13:14:58.987+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D4&quot;, &quot;c&quot;:&quot;ASIO&quot;, &#160; &#160; &quot;id&quot;:22596, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;startCommand&quot;,&quot;attr&quot;:{&quot;request&quot;:&quot;RemoteCommand 939 &amp;#8211; target:&lt;span class=&quot;error&quot;&gt;&amp;#91;cfg-1.internal:37001&amp;#93;&lt;/span&gt; db:config expDate:2024-01-22T13:15:28.987+00:00 cmd:{ find: \&quot;collections\&quot;, filter: &lt;/p&gt;
{ _id: /^db_20240119160851\\-800\\./ }
&lt;p&gt;, readConcern: { level: \&quot;majority\&quot;, afterOpTime: &lt;/p&gt;
{ ts: Timestamp(1705929298, 1), t: 2 }
&lt;p&gt; }, maxTimeMS: 30000 }&quot;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-22T13:14:58.989+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D4&quot;, &quot;c&quot;:&quot;ASIO&quot;, &#160; &#160; &quot;id&quot;:22596, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;startCommand&quot;,&quot;attr&quot;:{&quot;request&quot;:&quot;RemoteCommand 940 &amp;#8211; target:&lt;span class=&quot;error&quot;&gt;&amp;#91;cfg-1.internal:37001&amp;#93;&lt;/span&gt; db:config expDate:2024-01-22T13:15:28.989+00:00 cmd:{ find: \&quot;collections\&quot;, filter: &lt;/p&gt;
{ _id: /^db_20240119160851\\-801\\./ }
&lt;p&gt;, readConcern: { level: \&quot;majority\&quot;, afterOpTime: &lt;/p&gt;
{ ts: Timestamp(1705929298, 1), t: 2 }
&lt;p&gt; }, maxTimeMS: 30000 }&quot;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-22T13:14:58.991+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D4&quot;, &quot;c&quot;:&quot;ASIO&quot;, &#160; &#160; &quot;id&quot;:22596, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;startCommand&quot;,&quot;attr&quot;:{&quot;request&quot;:&quot;RemoteCommand 941 &amp;#8211; target:&lt;span class=&quot;error&quot;&gt;&amp;#91;cfg-2.internal:37001&amp;#93;&lt;/span&gt; db:config expDate:2024-01-22T13:15:28.991+00:00 cmd:{ find: \&quot;collections\&quot;, filter: &lt;/p&gt;
{ _id: /^db_20240119160851\\-802\\./ }
&lt;p&gt;, readConcern: { level: \&quot;majority\&quot;, afterOpTime: &lt;/p&gt;
{ ts: Timestamp(1705929298, 1), t: 2 }
&lt;p&gt; }, maxTimeMS: 30000 }&quot;}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-22T13:14:59.004+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;D4&quot;, &quot;c&quot;:&quot;ASIO&quot;, &#160; &#160; &quot;id&quot;:22596, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;startCommand&quot;,&quot;attr&quot;:{&quot;request&quot;:&quot;RemoteCommand 942 &amp;#8211; target:&lt;span class=&quot;error&quot;&gt;&amp;#91;cfg-3.internal:37001&amp;#93;&lt;/span&gt; db:config expDate:2024-01-22T13:15:29.004+00:00 cmd:{ find: \&quot;collections\&quot;, filter: &lt;/p&gt;
{ _id: /^db_20240119160851\\-803\\./ }
&lt;p&gt;, readConcern: { level: \&quot;majority\&quot;, afterOpTime: &lt;/p&gt;
{ ts: Timestamp(1705929298, 1), t: 2 }
&lt;p&gt; }, maxTimeMS: 30000 }&quot;}}&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;In our case, we are operating a cluster with more than 450k databases. Config servers is distributed between 3 sites:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Mongos &amp;lt;-&amp;gt; Config1 : 1ms&lt;/li&gt;
	&lt;li&gt;Mongos &amp;lt;-&amp;gt; Config2: 3ms&lt;/li&gt;
	&lt;li&gt;Mongos &amp;lt;-&amp;gt; Config3: 10ms&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;When the mongos start, as the queries as distributed equally to all config member, it takes approximately 30 minutes to start, which can be confirmed by doing the maths: `150k * 1ms + 150k * 3ms + 150k * 10ms = 35 minutes`&lt;/p&gt;

&lt;p&gt;We tried to set a lower `&#8211;localThreshold`, but it apparently only applies to communications between mongos and data shard.&lt;/p&gt;

&lt;p&gt;We are able to reproduce this behaviour by creating 25k on a testing env:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;it takes 2 minutes for a mongos to start when all config RS members are distributed between the three sites&lt;/li&gt;
	&lt;li&gt;it still takes 25s to start if config RS members are located on the same site&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;We were able to reproduce it on 5.0.24, but not on 6.0.13.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;4.4.22: 2 minutes&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;```&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-23T14:18:32.675+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &#160;&quot;c&quot;:&quot;CONTROL&quot;, &#160;&quot;id&quot;:23403, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;Build Info&quot;,&quot;attr&quot;:{&quot;buildInfo&quot;:{&quot;version&quot;:&quot;4.4.22&quot;,&quot;gitVersion&quot;:&quot;fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2&quot;,&quot;openSSLVersion&quot;:&lt;br/&gt;
&quot;OpenSSL 1.1.1f &#160;31 Mar 2020&quot;,&quot;modules&quot;:[],&quot;allocator&quot;:&quot;tcmalloc&quot;,&quot;environment&quot;:{&quot;distmod&quot;:&quot;ubuntu2004&quot;,&quot;distarch&quot;:&quot;x86_64&quot;,&quot;target_arch&quot;:&quot;x86_64&quot;}}}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-23T14:20:31.197+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &#160;&quot;c&quot;:&quot;NETWORK&quot;, &#160;&quot;id&quot;:23016, &#160; &quot;ctx&quot;:&quot;listener&quot;,&quot;msg&quot;:&quot;Waiting for connections&quot;,&quot;attr&quot;:{&quot;port&quot;:57000,&quot;ssl&quot;:&quot;off&quot;}}&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;5.0.24: 2 minutes&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;```&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-23T15:01:52.077+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &#160;&quot;c&quot;:&quot;CONTROL&quot;, &#160;&quot;id&quot;:23403, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;Build Info&quot;,&quot;attr&quot;:{&quot;buildInfo&quot;:{&quot;version&quot;:&quot;5.0.24&quot;,&quot;gitVersion&quot;:&quot;f034f0c51b3dffef4b8c9452d77ede9888f28f66&quot;,&quot;openSSLVersion&quot;:&lt;br/&gt;
&quot;OpenSSL 1.1.1f &#160;31 Mar 2020&quot;,&quot;modules&quot;:[],&quot;allocator&quot;:&quot;tcmalloc&quot;,&quot;environment&quot;:{&quot;distmod&quot;:&quot;ubuntu2004&quot;,&quot;distarch&quot;:&quot;x86_64&quot;,&quot;target_arch&quot;:&quot;x86_64&quot;}}}}&lt;/p&gt;
{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-23T15:03:53.645+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &#160;&quot;c&quot;:&quot;NETWORK&quot;, &#160;&quot;id&quot;:23016, &#160; &quot;ctx&quot;:&quot;listener&quot;,&quot;msg&quot;:&quot;Waiting for connections&quot;,&quot;attr&quot;:{&quot;port&quot;:57000,&quot;ssl&quot;:&quot;off&quot;}}&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;6.0.13: 7s&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;```&lt;/p&gt;

{&quot;t&quot;:\{&quot;$date&quot;:&quot;2024-01-23T15:53:10.296+00:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;, &#160;&quot;c&quot;:&quot;CONTROL&quot;, &#160;&quot;id&quot;:23403, &#160; &quot;ctx&quot;:&quot;mongosMain&quot;,&quot;msg&quot;:&quot;Build Info&quot;,&quot;attr&quot;:{&quot;buildInfo&quot;:{&quot;version&quot;:&quot;6.0.13&quot;,&quot;gitVersion&quot;:&quot;3b13907f9bdf6bd3264d67140d6c215d51bbd20c&quot;,&quot;openSSLVersion&quot;:&lt;br/&gt;
&quot;OpenSSL 1.1.1f &#160;31 Mar 2020&quot;,&quot;modules&quot;:[],&quot;allocator&quot;:&quot;tcmalloc&quot;,&quot;environment&quot;:{&quot;distmod&quot;:&quot;ubuntu2004&quot;,&quot;distarch&quot;:&quot;x86_64&quot;,&quot;target_arch&quot;:&quot;x86_64&quot;}}}}&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;Also, is it possible to backport the behaviour on 6.0.13 to upcoming 4.4 and 5.0 versions?&lt;/p&gt;

&lt;p&gt;Thank you!&lt;/p&gt;</description>
                <environment>Mongodb CE 4.4.27 - 5.0.24 - Ubuntu 22.04</environment>
        <key id="2554316">SERVER-85590</key>
            <summary>Mongos startup time increase linearly with the amount of databases in the cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="anthony@sendinblue.com">Anthony Brodard</reporter>
                        <labels>
                            <label>mongos</label>
                    </labels>
                <created>Tue, 23 Jan 2024 15:54:19 +0000</created>
                <updated>Mon, 5 Feb 2024 17:57:42 +0000</updated>
                            <resolved>Mon, 5 Feb 2024 17:57:42 +0000</resolved>
                                    <version>5.0 Required</version>
                    <version>4.4 Required</version>
                    <version>6.0 Required</version>
                                                    <component>Admin</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="6068013" author="eric.sedor" created="Mon, 5 Feb 2024 17:57:26 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=anthony%40sendinblue.com&quot; class=&quot;user-hover&quot; rel=&quot;anthony@sendinblue.com&quot;&gt;anthony@sendinblue.com&lt;/a&gt;, thanks for your patience. Specific improvements are backported where reasonable, but it&apos;s pretty unlikely that we&apos;ll backport to earlier versions the breadth of recent improvements for handling large numbers of namespaces. Hopefully moving to MongoDB 6.0 is an option for you!&lt;/p&gt;

&lt;p&gt;Sincerely,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 5 Feb 2024 17:57:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>anthony@sendinblue.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i39o6v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2rcv0:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;ol&gt;
	&lt;li&gt;Create an empty cluster&lt;/li&gt;
	&lt;li&gt;Restart a mongos, note the time diff between `&quot;msg&quot;:&quot;Build Info&quot;` and `&quot;msg&quot;:&quot;Waiting for connections&quot;` events&lt;/li&gt;
	&lt;li&gt;Create large batch of databases&lt;/li&gt;
	&lt;li&gt;Restart a mongos, note the time diff between `&quot;msg&quot;:&quot;Build Info&quot;` and `&quot;msg&quot;:&quot;Waiting for connections&quot;` events&lt;/li&gt;
&lt;/ol&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i39ac7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>