<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:49:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-60344] Action plan on lagging setFCV replicas breaking tests</title>
                <link>https://jira.mongodb.org/browse/SERVER-60344</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This ticket is to discuss and decide on proper actions on test breakages investigated by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cheahuychou.mao&quot; class=&quot;user-hover&quot; rel=&quot;cheahuychou.mao&quot;&gt;cheahuychou.mao&lt;/a&gt;:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The&#160;setFeatureCompatibilityVersion() replicates to &lt;a href=&quot;https://github.com/10gen/mongo/blob/a58ba9f4fcb62d0805d1e05512525be320299b4d/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L167&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;majority committed&lt;/a&gt;. It means when the command succeeds some replicas may still lag on the desired version.&lt;/li&gt;
	&lt;li&gt;When Mongos detects incompatible version is check-fails.&lt;/li&gt;
	&lt;li&gt;The Mongos connection pre-warm feature amplifies this race because now Mongos has a feature to pre-warm connections at startup, which includes non-primaries. A replica may still lag on set FCV() and causes Mongos to crash&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;There are several questions we need to resolve before addressing this.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Is set FCV() majority commit correct&lt;/b&gt; or we want to replicate to all replicas first?&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;My opinion it is correct. What an SRE wants from the system is that changing FCV is fast and durable. The condition when SRE performs upgrade and downgrade is usually not the case when it&apos;s a good time to reflex on complications when set FCV does not return or times out. What SRE wants is to get the confirmation and move forward with next actions.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Is it correct that Mongos pre-warms connections to non primaries?&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;My opinion: it might be tricky. Mongos doesn&apos;t know the primary, it gets the&#160;Grid::get() information and iterates over `shard.getHost()... getServers()` to pre-warm. Should it take additional wait time to find out just the primary, so make it 2-step procedure? Do we need to make the code that cumbersome?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The proposed action&lt;/b&gt;:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;make a best effort to find a simple solution to pre-warming the connections to the primary only.&lt;/li&gt;
	&lt;li&gt;If it gets unclean and/or requires extra overhead comparing to the existing code, just fix the tests.&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="1887026">SERVER-60344</key>
            <summary>Action plan on lagging setFCV replicas breaking tests</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="andrew.shuvalov@mongodb.com">Andrew Shuvalov</assignee>
                                    <reporter username="andrew.shuvalov@mongodb.com">Andrew Shuvalov</reporter>
                        <labels>
                    </labels>
                <created>Thu, 30 Sep 2021 15:14:04 +0000</created>
                <updated>Sun, 29 Oct 2023 21:47:59 +0000</updated>
                            <resolved>Wed, 13 Oct 2021 13:11:56 +0000</resolved>
                                                    <fixVersion>4.0.28</fixVersion>
                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4120455" author="xgen-internal-githook" created="Wed, 13 Oct 2021 02:34:43 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-60344&quot; title=&quot;Action plan on lagging setFCV replicas breaking tests&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-60344&quot;&gt;&lt;del&gt;SERVER-60344&lt;/del&gt;&lt;/a&gt; Mogos crash on incompatible mongod suspended during warm-up&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/30dc4f5d4bd98c4d2eb3d37644169b83dfcb5ede&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/30dc4f5d4bd98c4d2eb3d37644169b83dfcb5ede&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4095186" author="JIRAUSER1256988" created="Thu, 30 Sep 2021 19:56:26 +0000"  >&lt;p&gt;Based on the conversation above I&apos;ve made a change to simply temporarily suspend the crash during initial warm-up. Please review:&#160;&lt;a href=&quot;https://github.com/10gen/mongo/pull/1001&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/10gen/mongo/pull/1001&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4094857" author="renctan" created="Thu, 30 Sep 2021 17:56:45 +0000"  >&lt;p&gt;I wasn&apos;t part of the discussion when the project was laid out, but I do certainly agree with Lamont that it is that likely users would be using secondary connections. I checked the scope documents and it didn&apos;t explicitly say to pre-warm connections only to the primaries.&lt;/p&gt;</comment>
                            <comment id="4094776" author="JIRAUSER1256988" created="Thu, 30 Sep 2021 17:28:57 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=anton.oyung&quot; class=&quot;user-hover&quot; rel=&quot;anton.oyung&quot;&gt;anton.oyung&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt;&#160;- do you remember why it was decided to pre-warm connections to all ReplicaSet servers found in the connection string instead of finding just the primary? I spoke with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lamont.nelson&quot; class=&quot;user-hover&quot; rel=&quot;lamont.nelson&quot;&gt;lamont.nelson&lt;/a&gt;&#160;and his opinion was I should not touch this logic because a latency sensitive user might use reads from secondaries for performance reasons and limiting pre-warm to just primary defeats the purpose of this feature.&lt;/p&gt;</comment>
                            <comment id="4094772" author="vishnu.kaushik" created="Thu, 30 Sep 2021 17:28:24 +0000"  >&lt;p&gt;Just some thoughts I had when I initially diagnosed the BF on replication (copying &lt;a href=&quot;https://jira.mongodb.org/browse/BF-22699?focusedCommentId=4083135&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-4083135&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;my comment&lt;/a&gt; from there): &lt;em&gt;The mongos should be alright with viewing a minority of nodes that haven&apos;t reached a compatible wire version yet. I think the current method of fasserting immediately is a little bit extreme. The reason I say this is, for the sake of fault tolerance it is possible for the mongos to try to connect to a cluster where some minority of nodes are down. A minority of nodes being on the wrong FCV version is a less severe crime than them being down, so maybe the fassert is a little harsh.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I think the idea of finding the primary first is nice. The &lt;tt&gt;mongos&lt;/tt&gt; should be able to function fine so long as a majority of nodes in the shard are behaving as expected.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="976630">SERVER-44152</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 30 Sep 2021 17:28:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 17 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 17 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>145.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.shuvalov@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
            <customfieldvalue>vishnu.kaushik@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i03s7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hznjiv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5222">Sharding 2021-10-04</customfieldvalue>
    <customfieldvalue id="5223">Sharding 2021-10-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i03ecv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>