<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:25:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-51397] Mongos fails to serve requests for 30 secs when losing comm with one config replica set node</title>
                <link>https://jira.mongodb.org/browse/SERVER-51397</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Both ShardRegistry and CatalogCache lookups trigger a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard.cpp#L233&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Shard::exhaustiveFindOnConfig()&lt;/a&gt; that has a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard.cpp#L93&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;default 30s timeout&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Consider the following scenario:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;A mongos &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard_registry.cpp#L561&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;access its ShardRegistry producing a cache miss&lt;/a&gt; on the underlying ReadThroughCache.&lt;/li&gt;
	&lt;li&gt;A &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard_registry.cpp#L124&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ShardRegistry::lookup&lt;/a&gt; targeting the nearest config replica set node is started.&lt;/li&gt;
	&lt;li&gt;Communication with that specific config replica set node is lost due to network partition&lt;/li&gt;
	&lt;li&gt;The RSM marks the host as failed.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;All the subsequent requests that hit the same mongos, require access to the ShardRegistry, and arrive before the current lookup times out, they will all try to join the ongoing &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard_registry.cpp#L124&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ShardRegistry::lookup&lt;/a&gt; started at 2.&lt;/p&gt;

&lt;p&gt;All those requests will fail with `NetworkInterfaceExceededTimeLimit` as soon as the original lookup times out.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;In practice even if we have more then one config server replica set node and even if we are using the ReadPreference::nearest to fetch data from them. If we loose communication to one of them, it can happen that the mongos will not be able to serve any request for up to 30 secs.&lt;/p&gt;

&lt;p&gt;The same reasoning can be applied to the CatalogCache because it also builds on top of the ReadThroughCache and implements the lookups through the same &lt;a href=&quot;https://github.com/mongodb/mongo/blob/75eb9e18187b7853ed8e317825a2e282a54f92f0/src/mongo/s/client/shard.cpp#L233&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Shard::exhaustiveFindOnConfig()&lt;/a&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1503921">SERVER-51397</key>
            <summary>Mongos fails to serve requests for 30 secs when losing comm with one config replica set node</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="12300">Won&apos;t Do</resolution>
                                        <assignee username="backlog-server-sharding">[DO NOT USE] Backlog - Sharding Team</assignee>
                                    <reporter username="tommaso.tocci@mongodb.com">Tommaso Tocci</reporter>
                        <labels>
                    </labels>
                <created>Tue, 6 Oct 2020 17:32:54 +0000</created>
                <updated>Tue, 6 Dec 2022 02:06:46 +0000</updated>
                            <resolved>Thu, 22 Oct 2020 15:57:10 +0000</resolved>
                                    <version>4.4.1</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="3459422" author="kaloian.manassiev" created="Thu, 22 Oct 2020 16:00:36 +0000"  >&lt;p&gt;Due to the reliability of modern data centre networks, this situation is extremely unlikely to happen in practice and if it does it will go away after 30 seconds. Because of this it is not something we will invest time improving.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1505469">SERVER-51406</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="281610" name="primary_config_server_blackholed_from_mongos_mine.js" size="1358" author="tommaso.tocci@mongodb.com" created="Tue, 6 Oct 2020 18:30:56 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25141"><![CDATA[Sharding]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 22 Oct 2020 16:00:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 15 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 15 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>32.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-sharding</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyaw9z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxx9av:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;This can be reproduce easily with the following js test:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/281610/281610_primary_config_server_blackholed_from_mongos_mine.js&quot; title=&quot;primary_config_server_blackholed_from_mongos_mine.js attached to SERVER-51397&quot;&gt;primary_config_server_blackholed_from_mongos_mine.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;To make it fail consistently, the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a6da4c6162dfb47adfc17c76d27b31d8ec7ae087/src/mongo/s/catalog/sharding_catalog_client_impl.cpp#L91&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;kConfigReadSelector&lt;/a&gt; needs to be set to ReadPreference::Nearest.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyaijb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>