<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:58:40 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-41781] Configure shard to have deterministic failover in  sharding_multiple_ns_rs.js</title>
                <link>https://jira.mongodb.org/browse/SERVER-41781</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The test has one sharded collection and an unsharded collection. It inserts data into both, waits for replication to happen, and then kills the primary of the shard. The test fails while waiting for a new primary to be recognized; it verifies this by calling the &quot;connPoolStats&quot; command.&lt;/p&gt;

&lt;p&gt;The test gets the current primary by calling st.rs0.getPrimary() (which equals d21521 at this point) after the first one was killed. It waits for up to 5mins to verify that &quot;connPoolStats&quot; reflects this node as the new primary. The waiting started at 2019-06-06T16:07:22 (line 2368).&lt;/p&gt;

&lt;p&gt;According to the logs a new primary, d21521, was promoted at 16:07:20 (line 2183). However, this was not reflected in the output of the connPoolStats command. At 16:04:44 (line 3471), node d21522 decided to start a third election. This seems strange since there was a heartbeat between the two immediately before on line 3417. Regardless, d21522 won the election and was promoted to primary at 2019-06-06T16:07:44 (line 3605).&lt;/p&gt;

&lt;p&gt;The &quot;connPoolStats&quot; command reported d21520 (the original primary), as the primary up until 2019-06-06T16:07:50 (line 4210). After this, d21522 was returned as primary by this command. Since the test is still waiting for d21521 to become primary the test will ultimately fail after the timeout period.&lt;/p&gt;

&lt;p&gt;Since multiple elections can occur, this failure could be avoided by modifying awaitRSClientHosts with the ability to check who the current primary of the shard is multiple times &#8211; not just when the function is initially called. Perhaps, awaitRSClientHosts could accept either a string or a function to produce the current expected primary would work.&lt;/p&gt;</description>
                <environment></environment>
        <key id="803861">SERVER-41781</key>
            <summary>Configure shard to have deterministic failover in  sharding_multiple_ns_rs.js</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="lamont.nelson@mongodb.com">Lamont Nelson</assignee>
                                    <reporter username="lamont.nelson@mongodb.com">Lamont Nelson</reporter>
                        <labels>
                            <label>sharding-wfbf-day</label>
                    </labels>
                <created>Fri, 14 Jun 2019 21:43:23 +0000</created>
                <updated>Sun, 29 Oct 2023 22:19:49 +0000</updated>
                            <resolved>Thu, 24 Oct 2019 19:15:35 +0000</resolved>
                                                    <fixVersion>4.3.1</fixVersion>
                                    <component>Replication</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="2499776" author="xgen-internal-githook" created="Thu, 24 Oct 2019 19:25:01 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lamont Nelson&apos;, &apos;username&apos;: &apos;lamontnelson&apos;, &apos;email&apos;: &apos;lamont.nelson@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41781&quot; title=&quot;Configure shard to have deterministic failover in  sharding_multiple_ns_rs.js&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41781&quot;&gt;&lt;del&gt;SERVER-41781&lt;/del&gt;&lt;/a&gt; Make one node in replica set non-voting to prevent non-deterministic election in the test.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1137e076c4315f531ef77d2e70ad11370cb3872c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1137e076c4315f531ef77d2e70ad11370cb3872c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2493841" author="lamont.nelson" created="Mon, 21 Oct 2019 23:16:58 +0000"  >&lt;p&gt;Talked with Esha and decided to just do priority 0 here. The code review link is here: &lt;a href=&quot;https://mongodbcr.appspot.com/477540005/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mongodbcr.appspot.com/477540005/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2320234" author="vesselina.ratcheva" created="Wed, 10 Jul 2019 21:32:43 +0000"  >&lt;p&gt;To summarize the above, the shard needs to be configured to either only have two nodes or to give a priority of 0 to the third node.&lt;/p&gt;</comment>
                            <comment id="2320197" author="vesselina.ratcheva" created="Wed, 10 Jul 2019 21:04:28 +0000"  >&lt;p&gt;I&apos;ve had a bit more time to look at this now and I believe &lt;tt&gt;awaitRSClientHosts&lt;/tt&gt; does what it is intended to do from an API/function standpoint. We use it in our tests to verify that the specified host(s) are in the specified state (with agreement from mongos). The requirement is that caller knows what hosts and states they want. Making the function more lenient risks masking bugs (we have over 30 usages as of today). &lt;/p&gt;

&lt;p&gt;The real problem here is in the usage, or rather in the expectations of the test itself.  The shard has three nodes and no assignment of priorities between them. This by itself already makes failovers non-deterministic, as either of the other two nodes can get elected if one node goes down. It also opens up the possibility of multiple elections, as we have seen in this test - a flaky network and/or machine slowness can easily lead to this.  Line 3417 does not actually show a successful heartbeat - that is only the request. The response came in much later, on line 3506, and thus the node started another election in the interim. The test needs to configure the shard such that there is only one way for the failover to occur.&lt;/p&gt;

&lt;p&gt;We have historically similar situations many times with tests that we write on the replication team. Thankfully, we have over time gotten much better at configuring our tests so that we avoid this. We typically give a priority of 0 to the third node (which makes it completely unelectable), or take that node out of the picture altogether and only have two replica set members. Either of these solutions should work for this test. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 10 Jul 2019 21:04:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 15 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 15 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>lamont.nelson@mongodb.com</customfieldvalue>
            <customfieldvalue>vesselina.ratcheva@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv5dzb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvbaf3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3001">Repl 2019-07-01</customfieldvalue>
    <customfieldvalue id="3197">Sharding 2019-08-26</customfieldvalue>
    <customfieldvalue id="3199">Sharding 2019-09-23</customfieldvalue>
    <customfieldvalue id="3305">Sharding 2019-10-07</customfieldvalue>
    <customfieldvalue id="3306">Sharding 2019-10-21</customfieldvalue>
    <customfieldvalue id="3307">Sharding 2019-11-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;1. run sharding_multiple_ns_rs.js&lt;br/&gt;
2. trigger an additional election after the second primary is elected, but before connPoolStats returns up-to-date information&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hv508n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>