<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:04:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-43904] When stepping down, step up doesn&apos;t filter out frozen nodes</title>
                <link>https://jira.mongodb.org/browse/SERVER-43904</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;One of the recommended ways &lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt; to force a particular node to become primary is to freeze all non-candidate nodes and then call replSetStepDown on the primary.  As of MongoDB 3.6, that code attempts to step up a candidate (by calling replSetStepUp).  However, that code doesn&apos;t exclude frozen nodes, and attempting to step up a frozen node will simply fail (&quot;2019-10-09T00:24:05.517+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;conn352334&amp;#93;&lt;/span&gt; Not starting an election for a replSetStepUp request, since we are not electable due to: Not standing for election because I am still waiting for stepdown period to end at 2019-10-09T00:33:59.473+0000 (mask 0x20)&quot;).  This isn&apos;t particularly bad, since the unfrozen node will actually call for, and win, an election, but it does make failovers slower (up to electionTimeoutMillis slower, presumably).&lt;/p&gt;

&lt;p&gt;An alternative approach that we&apos;re using, that isn&apos;t explicitly documented, is to increase the priority of both the current and candidate node, and then run replSetStepDown.  I&apos;ve verified both in code and logs that this is effective at getting mongo to step up the candidate node consistently.  It might be nice to document this approach, since I think it offers improvements over both approaches currently mentioned.  Increasing the priority on just the candidate works, but tends to be slower since the &quot;priority takeover&quot; mechanism takes a few seconds to trigger, and provides less control than an explicit replSetStepDown.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="965249">SERVER-43904</key>
            <summary>When stepping down, step up doesn&apos;t filter out frozen nodes</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="xuerui.fa@mongodb.com">Xuerui Fa</assignee>
                                    <reporter username="bartle">David Bartley</reporter>
                        <labels>
                            <label>former-quick-wins</label>
                    </labels>
                <created>Wed, 9 Oct 2019 00:26:35 +0000</created>
                <updated>Sun, 29 Oct 2023 22:16:17 +0000</updated>
                            <resolved>Tue, 13 Oct 2020 17:17:23 +0000</resolved>
                                    <version>3.6.14</version>
                                    <fixVersion>4.9.0</fixVersion>
                    <fixVersion>4.0.23</fixVersion>
                    <fixVersion>4.4.4</fixVersion>
                    <fixVersion>4.2.13</fixVersion>
                                                        <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="3565668" author="xgen-internal-githook" created="Thu, 14 Jan 2021 18:57:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;XueruiFa&apos;, &apos;email&apos;: &apos;xuerui.fa@mongodb.com&apos;, &apos;username&apos;: &apos;XueruiFa&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43904&quot; title=&quot;When stepping down, step up doesn&amp;#39;t filter out frozen nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43904&quot;&gt;&lt;del&gt;SERVER-43904&lt;/del&gt;&lt;/a&gt;: Filter unelectable nodes during election handoff&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bf614cb57059c74830633855e28b3f4677cd4f8d)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/eb949a18023881d941e5367ff0ac6ccf4ae46b79&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/eb949a18023881d941e5367ff0ac6ccf4ae46b79&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3565213" author="xgen-internal-githook" created="Thu, 14 Jan 2021 16:14:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;XueruiFa&apos;, &apos;email&apos;: &apos;xuerui.fa@mongodb.com&apos;, &apos;username&apos;: &apos;XueruiFa&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43904&quot; title=&quot;When stepping down, step up doesn&amp;#39;t filter out frozen nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43904&quot;&gt;&lt;del&gt;SERVER-43904&lt;/del&gt;&lt;/a&gt;: Filter unelectable nodes during election handoff&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bf614cb57059c74830633855e28b3f4677cd4f8d)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fa5792bdc5408340489475b1f3bf5b8d709ebc20&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fa5792bdc5408340489475b1f3bf5b8d709ebc20&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3546827" author="xgen-internal-githook" created="Mon, 4 Jan 2021 15:46:11 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;XueruiFa&apos;, &apos;email&apos;: &apos;xuerui.fa@mongodb.com&apos;, &apos;username&apos;: &apos;XueruiFa&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43904&quot; title=&quot;When stepping down, step up doesn&amp;#39;t filter out frozen nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43904&quot;&gt;&lt;del&gt;SERVER-43904&lt;/del&gt;&lt;/a&gt;: Filter unelectable nodes during election handoff&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bf614cb57059c74830633855e28b3f4677cd4f8d)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/16dcdd77eb4f19fad01820612ffc16b1d3c15649&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/16dcdd77eb4f19fad01820612ffc16b1d3c15649&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3442484" author="xgen-internal-githook" created="Tue, 13 Oct 2020 16:14:42 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;XueruiFa&apos;, &apos;email&apos;: &apos;xuerui.fa@mongodb.com&apos;, &apos;username&apos;: &apos;XueruiFa&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43904&quot; title=&quot;When stepping down, step up doesn&amp;#39;t filter out frozen nodes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43904&quot;&gt;&lt;del&gt;SERVER-43904&lt;/del&gt;&lt;/a&gt;: Filter unelectable nodes during election handoff&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/bf614cb57059c74830633855e28b3f4677cd4f8d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/bf614cb57059c74830633855e28b3f4677cd4f8d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3020249" author="siyuan.zhou@10gen.com" created="Tue, 31 Mar 2020 22:49:40 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ying%40stripe.com&quot; class=&quot;user-hover&quot; rel=&quot;ying@stripe.com&quot;&gt;ying@stripe.com&lt;/a&gt; for the feedback. We&apos;ll keep this ticket to track the stepdown command&apos;s behavior with priorities and keep the general priority takeover issue in our radar.&lt;/p&gt;</comment>
                            <comment id="3020206" author="ying@stripe.com" created="Tue, 31 Mar 2020 22:06:26 +0000"  >&lt;p&gt;We are less concerned about the priority takeover because we don&apos;t use it. We tried it but it is worse than our current approach. It surprised us. We don&apos;t even know if it is a bug or expected behavior. If you think it is a bug, we can file a ticket.&#160;&lt;/p&gt;</comment>
                            <comment id="3020144" author="siyuan.zhou@10gen.com" created="Tue, 31 Mar 2020 21:27:48 +0000"  >&lt;blockquote&gt;&lt;p&gt;The use case for us is to fail over the primary to a dedicated server or a set of servers with minimal write unavailability.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I understand. My question is whether you are more concerned about the failover caused by planned maintenance or node failures. I understand you ran into this issue after a reconfig which seems a workaround of the undesired behavior cause by planned maintenance. Thus we are focusing on the planned maintenance in this ticket.&lt;/p&gt;

&lt;p&gt;If you are concerned about the general priority takeover behavior, could you please file a new ticket?&lt;/p&gt;</comment>
                            <comment id="3014201" author="ying@stripe.com" created="Mon, 30 Mar 2020 23:29:14 +0000"  >&lt;p&gt;The use case for us is to fail over the primary to a dedicated server or a set of servers with minimal write unavailability. In case it is not possible at a given time, we still want minimal write unavailability even if a different server is chosen as the primary.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3011776" author="siyuan.zhou@10gen.com" created="Sun, 29 Mar 2020 21:36:58 +0000"  >&lt;p&gt;I agree it is possible for a high priority node to fail to take over. However, since this ticket is about primary stepdown. The solution could be different. As &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=evin.roesle&quot; class=&quot;user-hover&quot; rel=&quot;evin.roesle&quot;&gt;evin.roesle&lt;/a&gt; mentioned before, the best way to get the fastest elections is to allow the system to decide which nodes should be elected on stepdown. One solution could be to attempt to choose the highest priority node to be the candidate on stepdown. That should be sufficient for planned maintenance using replSetStepDown command. Is that the major use case for you?&lt;/p&gt;

&lt;p&gt;Changing the behavior of priority takeover on general failover is a separate issue and worth a new ticket. If that&apos;s also you concern, could you please file a new ticket?&lt;/p&gt;</comment>
                            <comment id="3005519" author="bartle" created="Sat, 28 Mar 2020 03:16:50 +0000"  >&lt;p&gt;And tbc, while the original bug was indeed about replSetStepDown sometimes resulting in relatively long periods without a primary, the suggested fix was to use &quot;priority takeovers&quot;, but as noted above, this procedure also sometimes results in relatively long periods without a primary.  In short, both of the two suggested ways to do a failover sometimes result in relatively long periods without a primary.&lt;/p&gt;</comment>
                            <comment id="3005517" author="bartle" created="Sat, 28 Mar 2020 03:07:20 +0000"  >&lt;p&gt;In this case there was no explicit &quot;replSetStepDown&quot;; instead, we just increased priority on the intended primary and waited for it to become primary.  I guess it&apos;s not clear to me why this particular scenario shouldn&apos;t be expected?  Specifically, the following seems like a pretty common scenario:&lt;br/&gt;
1) Increase priority of node A&lt;br/&gt;
2) Node A calls for a dry-run election, which it wins, because (at the time) it&apos;s caught up to the current primary (node B)&lt;br/&gt;
3) Node A becomes slightly lagged (at this point there&apos;s still a primary accepting writes)&lt;br/&gt;
4) Node A calls for a real election (with incremented term), and that triggers the step down of node B (hence &quot;stepping down from primary, because a new term has begun&quot; in logs)&lt;br/&gt;
5) Node A doesn&apos;t win the election (because it&apos;s lagged, hence &quot;candidate&apos;s data is staler than mine&quot; in logs)&lt;/p&gt;

&lt;p&gt;The replset is now left without a primary.  Some new node (node C above) will eventually run for, and win, a future election, but this does increase the period during which there is no primary.&lt;/p&gt;

&lt;p&gt;In contrast, with an explicit replSetStepDown the old primary steps down, and then chooses an eligible secondary (i.e. one that is caught up and capable of being primary) to explicitly step up.  Since 1) writes will be blocked at the time of candidate selection, and 2) the candidate must be caught up, there&apos;s no possibility that this candidate will fail to be elected because it&apos;s lagged (but as mentioned originally, it can fail if it&apos;s frozen).&lt;/p&gt;</comment>
                            <comment id="3005505" author="siyuan.zhou@10gen.com" created="Sat, 28 Mar 2020 02:22:06 +0000"  >&lt;p&gt;Thanks for the log. As mentioned before, we also need the timeline of the stepdown command and any node stepping up after the stepdown but before node A running priority takeover. Priority takeover can only happen when the node knows of an existing primary, so there should be another primary before the priority takeover. This ticket is about stepdown, any solution will have to take that into consideration. That&apos;s why it&apos;s a key to understand the issue.&lt;/p&gt;</comment>
                            <comment id="3005331" author="ying@stripe.com" created="Fri, 27 Mar 2020 21:36:08 +0000"  >&lt;p&gt;Yes, we followed&#160;&#160;&quot;Force a Member to be Primary by Setting its Priority High&quot;. The following is the timeline of what happened. Node A is the node whose priority we increased, Node B is the old primary and Node C is the new primary.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Node A:&#160;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:14:57.157&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-26&amp;#93;&lt;/span&gt; Scheduling priority takeover at 2020-03-26T18:15:02.607+0000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.607&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-17&amp;#93;&lt;/span&gt; Canceling priority takeover callback&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.608&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-17&amp;#93;&lt;/span&gt; Starting an election for a priority takeover&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.608&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-19&amp;#93;&lt;/span&gt; dry election run succeeded, running for election in term 148&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Node B:&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.610&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;conn145984&amp;#93;&lt;/span&gt; stepping down from primary, because a new term has begun: 148&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.610&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-823&amp;#93;&lt;/span&gt; transition to SECONDARY from PRIMARY&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Node A:&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.610&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-29&amp;#93;&lt;/span&gt; VoteRequester(term 148) received a no vote from xxxxxx with reason &quot;candidate&apos;s data is staler than mine. candidate&apos;s last applied OpTime: { ts: Timestamp(1585246502, 85), t: 147 }, my last applied OpTime: { ts: Timestamp(1585246502, 86), t: 147 }&quot;; response message: { term: 148, voteGranted: false, reason: &quot;candidate&apos;s data is staler than mine. candidate&apos;s last applied OpTime: 
{ ts: Timestamp(1585246502, 85), t: 147 }
&lt;p&gt;, my last applied OpTime: { ts: Timest...&quot;, ok: 1.0, operationTime: Timestamp(1585246502, 86), $clusterTime: &lt;/p&gt;
{ clusterTime: Timestamp(1585246502, 86), signature: {...} }
&lt;p&gt; }&lt;/p&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:02.627&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-32&amp;#93;&lt;/span&gt; not becoming primary, we received insufficient votes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Node C:&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:07.808&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-71&amp;#93;&lt;/span&gt; Starting an election, since we&apos;ve seen no PRIMARY in the past 5000ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:07.808&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-71&amp;#93;&lt;/span&gt; conducting a dry run election to see if we could be elected. current term: 148&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:07.810&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-75&amp;#93;&lt;/span&gt; election succeeded, assuming primary role in term 149&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2020-03-26 11:15:07.810&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;replexec-75&amp;#93;&lt;/span&gt; transition to PRIMARY from SECONDARY&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3003714" author="siyuan.zhou@10gen.com" created="Fri, 27 Mar 2020 05:39:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ying%40stripe.com&quot; class=&quot;user-hover&quot; rel=&quot;ying@stripe.com&quot;&gt;ying@stripe.com&lt;/a&gt;, thanks for your feedback. Before exploring any solution, I&apos;d like to confirm your observation.&lt;/p&gt;

&lt;p&gt;By &quot;the documented approach&quot;, I assume you&apos;re referring to &quot;Force a Member to be Primary by Setting its Priority High&quot; on &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this page&lt;/a&gt;. Could you please clarify the timeline of the elections, especially when the stepdown runs? Did a node with lower priority steps up immediately after the stepdown? Did the highest priority node start the election afterwards? The timestamps when these events happened will help us verify the behavior.&lt;/p&gt;</comment>
                            <comment id="3003293" author="ying@stripe.com" created="Thu, 26 Mar 2020 21:33:22 +0000"  >&lt;p&gt;We tried the documented approach but got worse write availability. The problem is the candidate primary (with highest priority) may succeed during the dry run election and start a new election. Once a new election starts, the old primary steps down.&#160;But the candidate fails to get enough votes during the real election. So there is no primary for an extended period of time until it reaches election timeout and another node starts a new election.&lt;/p&gt;

&lt;p&gt;Can the candidate start a new election immediately if it fails to get enough votes?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2706788" author="bartle" created="Tue, 7 Jan 2020 05:34:03 +0000"  >&lt;p&gt;It&apos;s possible it&apos;s just a longer wait time, and not actually a longer write unavailability time.&lt;/p&gt;</comment>
                            <comment id="2541235" author="evin.roesle" created="Thu, 14 Nov 2019 18:32:41 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt;&#160;I&apos;m Evin and product manager on Aly&apos;s team.&lt;/p&gt;

&lt;p&gt;Often the best way to get the fastest elections is to allow the system to decide which nodes should be elected. This way we can avoid a large chunk of time in Primary Catchup. Primary Catchup is the phase of the election in which the elected node makes a best-effort attempt to replicate all current data before accepting new writes. This is why I&apos;m reluctant to add an option to stepdown that allows you to specify a candidate. I think this will lead to longer elections due to lack of insight about staleness.&#160;&lt;/p&gt;

&lt;p&gt;When you say that priority takeovers take longer when just specifying a higher priority on the candidate, this does not mean longer failover times with write unavailability but instead a longer wait time for the election to occur. Is that painful for you? The election time itself should be the same or even faster for priorityTakeovers because the node with the higher priority will wait until it is current in order to run, meaning the Primary Catchup phase does not take up any time minimizing write unavailability.&#160;&lt;/p&gt;

&lt;p&gt;Evin&lt;/p&gt;</comment>
                            <comment id="2491202" author="bartle" created="Sun, 20 Oct 2019 06:33:44 +0000"  >&lt;p&gt;I&apos;m trying to minimize failover duration, which (I assume) is what the &quot;step up&quot; feature was intended to do.  However, neither of the &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;documented&lt;/a&gt; approaches to electing a particular node would use that &quot;step up&quot; feature.  As stated above, it does not reliably work with replSetFreeze.  The other method (increase just the candidate primary&apos;s priority) does not involve an explicit replSetStepDown, so obviously the &quot;step up&quot; feature doesn&apos;t come into play.  In practice, both methods cause failovers that are a few seconds longer than the third approach I&apos;ve listed earlier (increase priority on both current and candidate primary before running replSetStepDown).&lt;/p&gt;

&lt;p&gt;If there&apos;s no plan to fix the poor interaction of replSetFreeze and replSetStepUp, I&apos;d suggest either removing mention of the &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/#procedures&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&quot;step up&quot; optimization&lt;/a&gt;, or document a third approach that would reliably work with the &quot;step up&quot; optimization.&lt;/p&gt;</comment>
                            <comment id="2490989" author="siyuan.zhou@10gen.com" created="Sat, 19 Oct 2019 04:18:13 +0000"  >&lt;p&gt;In that case, it sounds like you need priorities. Why didn&apos;t priority work for you?&lt;/p&gt;</comment>
                            <comment id="2490962" author="bartle" created="Sat, 19 Oct 2019 02:24:21 +0000"  >&lt;p&gt;Yes, adding an optional argument to replSetStepDown seems like it&apos;d simplify the existing approaches to failing over to a specific node.  Would you see much utility in allowing a list of candidates?  I guess one use case might be a situation where you&apos;re trying to do a cross-region failover, and don&apos;t particularly care which node is elected, as long as it&apos;s in the other region.&lt;/p&gt;</comment>
                            <comment id="2487606" author="siyuan.zhou@10gen.com" created="Thu, 17 Oct 2019 20:26:14 +0000"  >&lt;p&gt;We removed this behavior in the Election Handoff project. I&apos;d hope to avoid the complexity of adding it back. Besides, propagating via heartbeats is racy. Since the major use case of replSetFreeze is to choose a specific node to be the next primary, an alternative solution is to add an option to replSetStepDown command to nominate the next primary, so that stepdown will wait for the specific node instead.&lt;/p&gt;</comment>
                            <comment id="2476298" author="daniel.hatcher" created="Thu, 10 Oct 2019 18:54:45 +0000"  >&lt;p&gt;Thanks for the report. I&apos;ll pass this on to the Replication team to determine if there is an improvement to be made in regards to &quot;stepping up&quot; frozen nodes.&lt;/p&gt;</comment>
                            <comment id="2473168" author="bartle" created="Wed, 9 Oct 2019 03:00:44 +0000"  >&lt;p&gt;Specifically, frozen nodes seem to be considered unelectable, so you&apos;d expect &quot;e&quot; would be false.&lt;/p&gt;</comment>
                            <comment id="2473165" author="bartle" created="Wed, 9 Oct 2019 02:53:44 +0000"  >&lt;p&gt;Digging a bit into the code, it seems like frozenness is supposed to be communicated via the &quot;e&quot; field of heartbeats (under pv1 anyway), and the step-up code also seems to only consider electable candidates.  One possibility is that there&apos;s a race between nodes being frozen and the current primary discovering that, but even with a 10s sleep between those (i.e. 5x heartbeatMillis) I still observe (unsuccessful) step-ups to a frozen node.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="1512909">DOCS-13929</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1050777">SERVER-45116</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="947928">SERVER-43754</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>24.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 10 Oct 2019 18:54:45 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 3 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>I&amp;#39;m not sure if we track what heartbeats consist of, sorry if this doesn&amp;#39;t actually need downstream team attention!&lt;br/&gt;
&lt;br/&gt;
I added a heartbeat field, &amp;#39;electable&amp;#39;, to heartbeat responses. This tells the heartbeat response recipient if the node is electable to be primary or not. If a node has &amp;#39;electable&amp;#39; set to false, when the primary looks for a secondary to step up during election handoff, it will skip choosing that node as the new primary (since it is not electable)</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1816</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 3 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>bartle</customfieldvalue>
            <customfieldvalue>evin.roesle@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>xuerui.fa@mongodb.com</customfieldvalue>
            <customfieldvalue>ying@stripe.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvwr47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxw6a7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4311">Repl 2020-10-19</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvwddj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>