<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:06:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-66654] Failed election leaves recipient in unusable state</title>
                <link>https://jira.mongodb.org/browse/SERVER-66654</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;See build failure logs &lt;a href=&quot;https://evergreen.mongodb.com/lobster/build/03c7adfa8853c5ea811821bd059cc785/test/62898ff1be07c41f020b8802#bookmarks=0%2C34548&amp;amp;f~=100~j5%3Ars10&amp;amp;l=1&amp;amp;shareLine=7015&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Look for these specific lines:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[j5:rs10:n2] | 2022-05-22T01:21:24.036+00:00 I  COMMAND  21581   [conn44] &quot;Received replSetStepUp request&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[j5:rs10:n2] | 2022-05-22T01:21:24.036+00:00 D4 ELECTION 4615650 [conn44] &quot;Rescheduled election timeout callback&quot;,&quot;attr&quot;:{&quot;when&quot;:{&quot;$date&quot;:&quot;2022-05-23T03:20:58.441Z&quot;},&quot;requestedWhen&quot;:{&quot;$date&quot;:&quot;2022-05-23T01:21:24.036Z&quot;},&quot;error&quot;:{&quot;code&quot;:0,&quot;codeName&quot;:&quot;OK&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[j5:rs10:n2] | 2022-05-22T01:21:24.036+00:00 I  ELECTION 4615657 [conn44] &quot;Not starting an election for a replSetStepUp request, since we are not electable&quot;,&quot;attr&quot;:{&quot;reason&quot;:&quot;Not standing for election because I cannot see a majority (mask 0x1)&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[j5:rs10:n2] | 2022-05-22T01:21:24.036+00:00 I  COMMAND  21582   [conn44] &quot;replSetStepUp request failed&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot; :: caused by :: CommandFailed: Election failed.&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;When the logs reach that line, all recipient nodes are marked as SECONDARY, and subsequent attempts to &lt;a href=&quot;https://evergreen.mongodb.com/lobster/build/03c7adfa8853c5ea811821bd059cc785/test/62898ff1be07c41f020b8802#bookmarks=0%2C34548&amp;amp;f~=100~j5%3Ars10&amp;amp;l=1&amp;amp;shareLine=9303&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;clear the sync source, and find a new one&lt;/a&gt; seem to fail. Eventually the test times out &lt;a href=&quot;https://evergreen.mongodb.com/lobster/build/03c7adfa8853c5ea811821bd059cc785/test/62898ff1be07c41f020b8802#bookmarks=0%2C34548&amp;amp;f~=100~j5%3Ars10&amp;amp;l=1&amp;amp;shareLine=34510&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;waiting for a recipient primary&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2052578">SERVER-66654</key>
            <summary>Failed election leaves recipient in unusable state</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="didier.nadeau@mongodb.com">Didier Nadeau</assignee>
                                    <reporter username="matt.broadstone@mongodb.com">Matt Broadstone</reporter>
                        <labels>
                    </labels>
                <created>Mon, 23 May 2022 12:46:17 +0000</created>
                <updated>Thu, 18 Aug 2022 20:12:26 +0000</updated>
                            <resolved>Thu, 18 Aug 2022 20:12:26 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="4761788" author="steven.vannelli" created="Thu, 18 Aug 2022 20:12:26 +0000"  >&lt;p&gt;Team decided to close this out since it is low impact and hasn&apos;t happened in a while. Will address via a new BF if it arises again.&#160;&lt;/p&gt;</comment>
                            <comment id="4592082" author="JIRAUSER1262830" created="Fri, 3 Jun 2022 03:10:07 +0000"  >&lt;p&gt;Another occurence happened while running a patch for the shard split passthrough : &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_all_feature_flags_required_shard_split_stepdown_jscore_passthrough_1_enterprise_patch_f2d1b7432de2c2c06b03d557bdab3f863ef6708d_6298f4bd3066156f51292bed_22_06_02_17_35_12?execution=0&amp;amp;sortBy=STATUS&amp;amp;sortDir=ASC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;task&lt;/a&gt;. We can see in &lt;a href=&quot;https://evergreen.mongodb.com/lobster/build/fc2dedad04bb24b9dfc4fdcc00f3f51d/test/6298fcb790413047bce3b1d2#bookmarks=0%2C4499%2C4500%2C4504%2C4533%2C4534%2C4551%2C4577%2C4580%2C4639%2C47293&amp;amp;l=1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the annoted log&lt;/a&gt; that it failed due to a timeout to get the recipient primary, and that election had failed before that because two nodes see the all other nodes as DOWN.&lt;/p&gt;

&lt;p&gt;When we look at log for `j0:rs11:n0`, we see the same issue where an out of sync hearbeat set a node as down even though it has applied the new config (and a more recent hearbeat received before informed us of that). See L4499-4500 and L4533-4534 that confirm a response to an older hearbeat request (67) is received after a newer (request 69), and that mistakenly sets the node as down.&lt;/p&gt;</comment>
                            <comment id="4583195" author="JIRAUSER1262830" created="Tue, 31 May 2022 14:53:26 +0000"  >&lt;p&gt;It turns out the nodes do not get stuck in state DOWN : I was looking for string &quot;is now in state&quot; to check for MemberData state change, but the actual string is &quot;Member is in new state&quot; (I confused the two logs formats in LOGV2 &lt;a href=&quot;https://github.com/10gen/mongo/blob/master/src/mongo/db/repl/member_data.cpp#L73&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;). The nodes were seeing each other as DOWN during the replSetStepUp, which explains why it failed. Then the states were updated and an election would have succeeded. However the election never happened as the default timeout for tests run on evergreen is 86400000s. The test timed out as an election never happened.&lt;/p&gt;</comment>
                            <comment id="4581730" author="JIRAUSER1262830" created="Mon, 30 May 2022 19:18:38 +0000"  >&lt;p&gt;The problem appears to come from recipient nodes marking each other as DOWN during the application of the recipient config. The nodes that are DOWN are never reset (we don&apos;t see the log accompanying the reset) and they are not reset during the recipient config application either. The nodes keep exchanging heartbeat until the end, seemingly without issue, but they don&apos;t seem to clear the DOWN attribute (it seems like it should be reset &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/topology_coordinator.cpp#L1112&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; but it never happens), which means none of them can ever be elected.&lt;/p&gt;

&lt;p&gt;The next step would, ideally, be to understand why DOWN is never cleared and confirm the cause by reproducing the issue.&lt;/p&gt;

&lt;p&gt;We should be able to fix this by modifying the condition to mark a node as DOWN. It might also require to reset the nodes states when applying a recipient config.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 30 May 2022 19:18:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 24 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2357</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>steven.vannelli@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 24 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>didier.nadeau@mongodb.com</customfieldvalue>
            <customfieldvalue>matt.broadstone@mongodb.com</customfieldvalue>
            <customfieldvalue>steven.vannelli@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0vvev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hragmk:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="6268">Server Serverless 2022-06-13</customfieldvalue>
    <customfieldvalue id="6269">Server Serverless 2022-06-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0vhk7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>