<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:00:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42602] Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.</title>
                <link>https://jira.mongodb.org/browse/SERVER-42602</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;There are 2 kinds of phases in rollback fuzzer test suites.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;State Transition Phase - RollbackTest transitions to predefined state before the rollback fuzzer gets into workload execution phase.&lt;/li&gt;
	&lt;li&gt;Workload Execution Phase - rollback fuzzer executes some list of random commands (including restartNode cmd which can result in change of primary) on the replica set.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;After the RollbackTest transitions to &quot;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L362-L371&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transitionToSyncSourceOperationsDuringRollback&lt;/a&gt;&quot; state, we break the assumption mentioned &lt;a href=&quot;https://github.com/10gen/jstestfuzz/blob/039282007e0144fe1fcb5b8da9e9906332bf2dcf/src/fuzzers/rollback/rollback_runner_lib.js#L168-L170&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here &lt;/a&gt; in rollback fuzzer. After the &quot;transitionToSyncSourceOperationsDuringRollback&quot; state, the topology looks like below.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;CurSecondary/Node to be rolled back&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &#160; &#160; &#160; &#160; |&lt;br/&gt;
 &#160; &#160; &#160; &#160; |&lt;br/&gt;
 &#160; &#160; &#160; &#160; |&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;CurPrimary&amp;#93;&lt;/span&gt;-------- &lt;span class=&quot;error&quot;&gt;&amp;#91;TieBreakerNode&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Once the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L369&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;curSecondary&lt;/a&gt; node gets rolled back successfully (i.e) caught up to curPrimary, restarting a curPrimary can result in curSecondary to become the new primary. As a result, during workload execution phase, unconditional step down can happen due to slow planned node restarts (i.e. node restarts taking long time). And, that&#160; leads to undesired behavior in&#160; rollback_fuzzer_&lt;span class=&quot;error&quot;&gt;&amp;#91;un&amp;#93;&lt;/span&gt;clean_shutdown suites. So, in order to fix the issue, we should have below 2 contracts.&lt;/p&gt;

&lt;p&gt;1)&#160;During workload execution phase, unconditional step down can happen only due to some transient network issues and not because of slow planned node restarts (i.e. node restarts taking long time).&lt;/p&gt;

&lt;p&gt;2) Restarting nodes by rollback fuzzer can change the original primary only if all the 3 nodes are connected.&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="882812">SERVER-42602</key>
            <summary>Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="suganthi.mani@mongodb.com">Suganthi Mani</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Fri, 2 Aug 2019 06:39:37 +0000</created>
                <updated>Sun, 29 Oct 2023 22:18:26 +0000</updated>
                            <resolved>Fri, 30 Aug 2019 17:18:43 +0000</resolved>
                                                    <fixVersion>4.2.1</fixVersion>
                    <fixVersion>4.3.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>1</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2402776" author="xgen-internal-githook" created="Sat, 31 Aug 2019 13:19:09 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;smani87&apos;, &apos;email&apos;: &apos;suganthi.mani@mongodb.com&apos;, &apos;name&apos;: &apos;Suganthi Mani&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42602&quot; title=&quot;Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42602&quot;&gt;&lt;del&gt;SERVER-42602&lt;/del&gt;&lt;/a&gt; retry replSetFreeze on rollback node in tests if replica set config has not been loaded&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 6f308bbc6f495da46029b6e6316189a14e7842a3)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fef676084abb58b23103b03c2ef7cc05c3cfb023&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fef676084abb58b23103b03c2ef7cc05c3cfb023&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2402738" author="xgen-internal-githook" created="Sat, 31 Aug 2019 10:30:52 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Benety Goh&apos;, &apos;username&apos;: &apos;benety&apos;, &apos;email&apos;: &apos;benety@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42602&quot; title=&quot;Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42602&quot;&gt;&lt;del&gt;SERVER-42602&lt;/del&gt;&lt;/a&gt; retry replSetFreeze on rollback node in tests if replica set config has not been loaded&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/6f308bbc6f495da46029b6e6316189a14e7842a3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/6f308bbc6f495da46029b6e6316189a14e7842a3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2402194" author="xgen-internal-githook" created="Fri, 30 Aug 2019 17:32:50 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;suganthi.mani@mongodb.com&apos;, &apos;name&apos;: &apos;Suganthi Mani&apos;, &apos;username&apos;: &apos;smani87&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42602&quot; title=&quot;Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42602&quot;&gt;&lt;del&gt;SERVER-42602&lt;/del&gt;&lt;/a&gt; Guarantees that the unconditional step down does not happen due to slow node restarts in rollback_fuzzer_&lt;span class=&quot;error&quot;&gt;&amp;#91;un&amp;#93;&lt;/span&gt;clean_shutdowns suites.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 2cba3bf3640aed0121a05f6396cedadd10a06880)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fd2ce602d86b858d748ca7c40f2a96a77a1175aa&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fd2ce602d86b858d748ca7c40f2a96a77a1175aa&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2402032" author="xgen-internal-githook" created="Fri, 30 Aug 2019 16:10:31 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Suganthi Mani&apos;, &apos;username&apos;: &apos;smani87&apos;, &apos;email&apos;: &apos;suganthi.mani@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42602&quot; title=&quot;Guarantee that unconditional step down will not happen due to slow node restarts in  rollback_fuzzer_[un]clean_shutdowns suites.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42602&quot;&gt;&lt;del&gt;SERVER-42602&lt;/del&gt;&lt;/a&gt; Guarantees that the unconditional step down does not happen due to slow node restarts in rollback_fuzzer_&lt;span class=&quot;error&quot;&gt;&amp;#91;un&amp;#93;&lt;/span&gt;clean_shutdowns suites.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2cba3bf3640aed0121a05f6396cedadd10a06880&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2cba3bf3640aed0121a05f6396cedadd10a06880&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2363661" author="suganthi.mani" created="Thu, 8 Aug 2019 21:14:50 +0000"  >&lt;p&gt;I totally forgot about&#160;replSetFreeze cmd, that&apos;s really a good solution. transitionToSyncSourceOperationsDuringRollback() can execute {replSetFreeze: 24*60*60 /* 24 hrs */} and we can unfreeze it during transitionToSteadyStateOperations() by running&#160; {replSetFreeze: 0}.&lt;/p&gt;</comment>
                            <comment id="2361877" author="judah.schvimer" created="Wed, 7 Aug 2019 19:50:32 +0000"  >&lt;p&gt;I also prefer solution 1. Instead of adding a failpoint though, can we use the &lt;tt&gt;replSetFreeze&lt;/tt&gt; command with an &quot;infinite&quot; or 0 timeout to accomplish the same goal? This makes use of things users could actually do and that we already have which I like.&lt;/p&gt;</comment>
                            <comment id="2360094" author="suganthi.mani" created="Tue, 6 Aug 2019 18:49:10 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; for the detailed explanation. After discussing with him, we decided to  drop solution #2 since reconfig correctness reasoning is not straightforward and adds more complexity to the rollback test fixture.&lt;/p&gt;</comment>
                            <comment id="2359730" author="suganthi.mani" created="Tue, 6 Aug 2019 15:51:33 +0000"  >&lt;p&gt;Currently, we have 2 solutions to enforce the contract.&lt;/p&gt;

&lt;p&gt;1) Introduce a new failpoint like &quot;FailCandidateToBecomeElectable&quot; which &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L879-L906&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;prevents a node from running election&lt;/a&gt;. So, turn this fail point on during&#160;&quot;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L362-L371&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transitionToSyncSourceOperationsDuringRollback&lt;/a&gt;&quot; state. As a result, during workload execution phase, any curPrimary restarts after the rollback event (i.e.&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L369&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;curSecondary&lt;/a&gt;&#160;successfully rolled back) will not change the original primary. And, then, turn the failpoint off during &quot;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L206-L240&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transitionToSteadyStateOperations&lt;/a&gt;&quot; state.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;This means, we should also make sure when the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L369&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;curSecondary&lt;/a&gt;&#160;is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L390&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;restarted,&lt;/a&gt;&#160;we should reactivate the failpoint on. Else, the curSecondary can run for election.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;2) Other solution is that, setting the priority value of curSecondary to 0 using reconfig cmd which would prevent curSecondary from running election. During &quot;transitionToSyncSourceOperationsDuringRollback&quot; execute the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L364&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;replSetReconfig&lt;/a&gt; cmd on primary with the change in the curSecondary priority value to 0.&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L364&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;replSetReconfig&lt;/a&gt;&#160; on curPrimary make sure that the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/src/mongo/db/repl/replication_coordinator_impl.cpp#L2621&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;new config is persisted locally&lt;/a&gt;. So, any subsequent&#160; &quot;replSetRequestVotes&quot; request from curSecondary with previous stale config will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/src/mongo/db/repl/topology_coordinator.cpp#L2718-L2724&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fail&lt;/a&gt;. And, then reset the priority value of curSecondary to 1 during &quot;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/fbe6b6601738a38a38f3387fb336a1e1284f6b88/jstests/replsets/libs/rollback_test.js#L206-L240&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transitionToSteadyStateOperations&lt;/a&gt;&quot; state.&lt;/p&gt;</comment>
                            <comment id="2359669" author="william.schultz" created="Tue, 6 Aug 2019 15:31:30 +0000"  >&lt;p&gt;To follow up on Suganthi&apos;s description, below is an attempt to summarize some of our general discussion and reasoning about each state of the RollbackTest fixture. Note that server replication is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/134a4083953270e8a11430395357fb70a29047ad/jstests/replsets/libs/rollback_test.js#L81&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stopped&lt;/a&gt; on the tiebreaker node throughout the whole process, until the final transition to steady state operations. We looked at each state and tried to convince ourselves if shutdowns may or may not cause unconditional stepdowns in RollbackTest. We assume that we always wait for a new, stable primary after shutting down a node. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt; can verify the reasoning below if anything seems incorrect. If nothing else, hopefully this helps clarify each phase of RollbackTest, for future reference.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSteadyStateOps&lt;/b&gt;:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /   \&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;P1 -  S&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Shutdowns may occur on either node and switch the primary an arbitrary number of times. At the &lt;br/&gt;
 end of this phase we just need all nodes to be connected and to have some stable primary. We don&apos;t &lt;br/&gt;
 need to worry who the primary is, though.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kRollbackOps&lt;/b&gt;:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /    &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;P1    S&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;If we shut down either the current primary or secondary, the partition should cause each node to return to the same state since the rollback node (P1) will eventually get re-elected. After this phase we need the oplog of the current primary (the node that will roll back) to be diverged from the sync source.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSyncSourceOpsBeforeRollback&lt;/b&gt;:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;     \&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;P1    P2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;If we shut down either node, the sync source node (P2) will end up being re-elected as long as we wait for a new primary after each shutdown. The divergent rollback node should never get elected since it is isolated. It will remain either as a stale primary or a secondary with a divergent oplog. After this phase completes we assume the sync source node is primary and has applied some operations in its new term.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSyncSourceOpsDuringRollback&lt;/b&gt;:&lt;/p&gt;

&lt;p&gt;(&lt;em&gt;before rollback completion&lt;/em&gt;)&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;     \&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;R  -  P2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;If we shut down the sync source node during this state it will be able to start up and get re-elected eventually since the rollback node has a stale/divergent oplog. If we shut down the rollback node before or during its rollback it will still have a divergent oplog and eventually end up back in rollback again.&lt;/p&gt;

&lt;p&gt;(&lt;em&gt;after rollback completion&lt;/em&gt;)&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;     \&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;S  -  P2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Now that both nodes have the same oplog, it is possible that either can get elected, so shutdown of either one could lead to arbitrary switching of the primary. Additionally, arbitrarily slow restarts could cause a current primary (e.g. S, if it got elected) to step down unconditionally due to a liveness timeout, since it does not necessarily have the support of the tiebreaker node given the network topology. This is the problem outlined in the ticket description above.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;kSteadyStateOps&lt;/b&gt;:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   T&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /   \&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;S  -  P2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;The original state.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="885460">SERVER-42650</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="920861">SERVER-43237</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 6 Aug 2019 15:31:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 23 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 23 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hviqm7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hv7oov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3200">Repl 2019-08-26</customfieldvalue>
    <customfieldvalue id="3201">Repl 2019-09-09</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvicvj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>