<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:26:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31274] With chaining enabled, a healthy primary may lose an election to a node that does not see it</title>
                <link>https://jira.mongodb.org/browse/SERVER-31274</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Consider the following example with PV1,&lt;/p&gt;

&lt;p&gt;A three node replica set with chaining enabled where A is primary and C has B as it&apos;s sync source: A &amp;lt;- B &amp;lt;- C&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/44d011a20109ef8769d881306e33bc92a2b44f4d/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L823&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;If B does not see heartbeats from A, it will call for an election&lt;/a&gt;. C will vote for B &lt;a href=&quot;https://github.com/mongodb/mongo/blob/78f2325fbf771242457315819df2e5cb341eaeee/src/mongo/db/repl/topology_coordinator_impl.cpp#L3079-L3134&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;despite having a healthy connection to A&lt;/a&gt;. As a result, a failover will occur despite a majority of nodes being able to see the primary.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/166494/166494_Heartbeats.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="433249">SERVER-31274</key>
            <summary>With chaining enabled, a healthy primary may lose an election to a node that does not see it</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="kelsey.schubert@mongodb.com">Kelsey Schubert</reporter>
                        <labels>
                    </labels>
                <created>Tue, 26 Sep 2017 20:49:05 +0000</created>
                <updated>Fri, 27 Oct 2023 13:54:13 +0000</updated>
                            <resolved>Mon, 2 Oct 2017 22:57:48 +0000</resolved>
                                    <version>3.2.0</version>
                    <version>3.4.0</version>
                    <version>3.5.13</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1691255" author="henrik.ingo@10gen.com" created="Fri, 6 Oct 2017 11:08:46 +0000"  >&lt;p&gt;That&apos;s actually exactly what I meant with my comment. As you point out, distributed systems will always have races like this, and trying to avoid them might often lead in the wrong direction (longer failover time / worst of both worlds). However, it may be possible to adjust code and/or parameters so that one outcome is likely to happen most of the time.&lt;/p&gt;</comment>
                            <comment id="1690920" author="spencer" created="Thu, 5 Oct 2017 21:50:21 +0000"  >&lt;p&gt;While researching the code to prepare my previous comment I noticed a potential optimization that could help C change sync sources sooner and narrow the window in which the unnecessary failover will occur.  I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31429&quot; title=&quot;Use the last fetched instead of the last applied to evaluate sync sources&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31429&quot;&gt;&lt;del&gt;SERVER-31429&lt;/del&gt;&lt;/a&gt; for that optimization.  It won&apos;t eliminate the issue entirely, but might make it harder to hit.&lt;/p&gt;</comment>
                            <comment id="1690903" author="spencer" created="Thu, 5 Oct 2017 21:45:22 +0000"  >&lt;p&gt;Yep, I think that&apos;s a good description.  The thing is, we already have code to cause C to start syncing from A.  When a node is caught up to its current sync source, and its current sync source doesn&apos;t have a sync source, we change sync sources: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/cb9f7cdcb7eb6ad9077ac8af3a4c0d7275c7e34f/src/mongo/db/repl/topology_coordinator_impl.cpp#L3015&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/cb9f7cdcb7eb6ad9077ac8af3a4c0d7275c7e34f/src/mongo/db/repl/topology_coordinator_impl.cpp#L3015&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The problem is that that check fundamentally races with the election timeout on B, so if B&apos;s election timeout fires before C decides to switch sync sources, then the failover will happen.  B is going to call for an election no matter what once it determines A to be down for longer than its election timeout, so to avoid the failover that election needs to fail because C is already replicating from A and is now ahead of B.  For that to happen, B needs to notice that it can no longer sync from A, C needs to catch up to B (and in the process learn that B has no sync source), C needs to go through the sync source selection process and choose A as it&apos;s new source, and C needs to fetch and apply a new op from A.  So the chances of B calling for an election before all that happens is decently high, and the only way to prevent that is to make B take longer to win its election, which will slow down all failovers.&lt;/p&gt;</comment>
                            <comment id="1690023" author="henrik.ingo@10gen.com" created="Thu, 5 Oct 2017 10:51:05 +0000"  >&lt;p&gt;Just an additional comment to elaborate on the trade off in play, to see if I follow your discussion correctly?&lt;/p&gt;

&lt;p&gt;Since C is syncing from B, if B cannot sync from A, then both secondaries are in fact disconnected from A. For example, w:majority writes will not succeed at this point. Possible ways to fix this are:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;New election as in the ticket description&lt;/li&gt;
	&lt;li&gt;C starts syncing from A&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I think you&apos;re on the right path when you say that the goal is to minimize failover time. Whatever can be done to get ops succeeding again, as quickly as possible, is the right solution.&lt;/p&gt;

&lt;p&gt;For the first alternative, there will be an additional cost when clients have to discover and reconnect to the newly elected primary B. As such, it could actually be beneficial to go the second path, C starts syncing from A. To know the right answer, we would have to do actual measurements. (And in any case, you are in a much better position than myself to reason about what the code is doing here.)&lt;/p&gt;

&lt;p&gt;It&apos;s quite likely though, that &quot;whoever happens to react first&quot; is in fact the best, or at least a good choice. As such, changes to the current behavior would at best be low priority.&lt;/p&gt;</comment>
                            <comment id="1687241" author="spencer" created="Mon, 2 Oct 2017 22:57:48 +0000"  >&lt;p&gt;There&apos;s no way to improve this without slowing down all failovers&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="333037">SERVER-27125</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="441395">SERVER-31429</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="166494" name="Heartbeats.png" size="100145" author="kelsey.schubert@mongodb.com" created="Tue, 26 Sep 2017 20:49:52 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 2 Oct 2017 22:57:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 18 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 18 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>henrik.ingo@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htfs8n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht7kxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htfebz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>