<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39367] lastOpCommitted being reset on restart can cause sync source cycle</title>
                <link>https://jira.mongodb.org/browse/SERVER-39367</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Consider two nodes in a set, nodes A and B. Node A restarts (resetting its lastOpCommitted to 0) and is ahead of node B. Node B syncs from A. A has no lastOpCommitted so A chooses to sync from B which has a non-zero lastOpCommitted, even though B is already syncing from A.&lt;/p&gt;</description>
                <environment></environment>
        <key id="683456">SERVER-39367</key>
            <summary>lastOpCommitted being reset on restart can cause sync source cycle</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="siyuan.zhou@mongodb.com">Siyuan Zhou</assignee>
                                    <reporter username="judah.schvimer@mongodb.com">Judah Schvimer</reporter>
                        <labels>
                    </labels>
                <created>Mon, 4 Feb 2019 19:06:39 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:26 +0000</updated>
                            <resolved>Wed, 27 Feb 2019 01:53:03 +0000</resolved>
                                                    <fixVersion>4.1.9</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="2181785" author="william.schultz" created="Fri, 15 Mar 2019 15:15:29 +0000"  >&lt;p&gt;To clarify, the multi node cycle repro referenced above is based on the original protocol where we propagate commit points only via the sync source spanning tree, and where we allow choosing a sync source with a higher commit point (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;). It does not use the technique of propagating commit points via heartbeats, which is the new fix proposal implemented in this ticket. The intent was to demonstrate a more general case of the original bug (a 2 node cycle) brought up in this ticket.&lt;/p&gt;</comment>
                            <comment id="2179118" author="william.schultz" created="Tue, 12 Mar 2019 23:51:42 +0000"  >&lt;p&gt;Here is a &lt;a href=&quot;https://gist.github.com/will62794/98af781424e81931dbd496523c7d81b1#file-multi_node_sync_cycle_appendentryandlearncommitpointfromsyncsource-trace&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;link&lt;/a&gt; to the raw trace. To summarize the essence of it though, consider the following progression in a 4 node replica set (the values that change in each step are highlighted in &lt;font color=&quot;#de350b&quot;&gt;red&lt;/font&gt;):&lt;/p&gt;

&lt;p&gt;&lt;b&gt;State 1 (Initial State)&lt;/b&gt;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;index&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;2&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&#160;&#160;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n1&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;commitPoint=2&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;syncSource=none&lt;/font&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n2&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n1&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n3&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n1&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n4&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n2&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&lt;b&gt;State 2&lt;/b&gt;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&lt;b&gt;index&lt;/b&gt;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;2&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&lt;b&gt;n1&lt;/b&gt;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&#160;&#160;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;commitPoint=2&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;syncSource=none&lt;/font&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&lt;b&gt;n2&lt;/b&gt;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;commitPoint=0&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&lt;b&gt;n3&lt;/b&gt;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#de350b&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&#160;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#de350b&quot;&gt;commitPoint=2&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;syncSource=n1&lt;/font&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;&lt;b&gt;n4&lt;/b&gt;&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#172b4d&quot;&gt;commitPoint=0&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&lt;b&gt;State 3&lt;/b&gt;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;index&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n1&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n2&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n3&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#de350b&quot;&gt;syncSource=n4&lt;/font&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n4&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&lt;b&gt;State 4&lt;/b&gt;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;index&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n1&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n2&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n3&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#de350b&quot;&gt;1&lt;/font&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n4&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;&lt;b&gt;State 5&lt;/b&gt;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;index&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n1&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n2&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;font color=&quot;#de350b&quot;&gt;syncSource=n3&lt;/font&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n3&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;n4&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;commitPoint=0&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;syncSource=n2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;It should be clear that State 1 is an easy state to end up in by simple steady state operation of the protocol. In State 2, n3 syncs an entry from n1 and advances its commit point. In State 3, n3 then decides to switch sync sources and sync from n4, which is legitimate, since n4&apos;s log is newer than n2. In State 4, n3 then replicates a new log entry from n4, so that the logs of all nodes are now the same. Then, in State 5, n2 decides to switch sync sources and sync from n3, which is legitimate, since the commit point of n3 is higher than its own, even though their logs are the same. This creates the sync cycle of n2-&amp;gt;n3-&amp;gt;n4-&amp;gt;n2. I was not able to reproduce a similar case in a 3 node replica set.&#160;&lt;/p&gt;</comment>
                            <comment id="2178830" author="siyuan.zhou@10gen.com" created="Tue, 12 Mar 2019 19:35:02 +0000"  >&lt;p&gt;Using TLC model checker, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; found that it&apos;s possible to form a sync source cycle with more than 2 nodes. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt;, could you please post the TLC trace here? It can only happen in a replset of at least 4 nodes, right?&lt;/p&gt;

&lt;p&gt;&lt;del&gt;I&apos;m asking because detecting 2-node cycle in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a78f546ff0caab8c33738aa922a294d9143a5397/src/mongo/db/repl/topology_coordinator.cpp#L2542&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shouldChangeSyncSource()&lt;/a&gt; is trivial since we know the sync source of our sync source. This might be a potential workaround in backport to avoid cycle in mixed-version replset.&lt;/del&gt;&lt;/p&gt;

&lt;p&gt;Never mind about the backport, a multi-node cycle could happen more frequently in mixed-version replsets.&lt;/p&gt;</comment>
                            <comment id="2164619" author="tess.avitabile" created="Wed, 27 Feb 2019 14:03:17 +0000"  >&lt;p&gt;As I recall, we agreed that we would not re-enable chaining in the rollback test fixture. Although it was valuable to catch this sync source cycle bug, the intention of that fixture is not to test for sync source cycles, and enabling chaining in the fixture requires us to be very careful about when replication is enabled on the tiebreaker node. I think we can close &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39497&quot; title=&quot;Revert SERVER-33248&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39497&quot;&gt;&lt;del&gt;SERVER-39497&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2163848" author="siyuan.zhou@10gen.com" created="Wed, 27 Feb 2019 01:56:14 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, my patch didn&apos;t revert &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39353&quot; title=&quot;disable chaining in rollback_test.js&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39353&quot;&gt;&lt;del&gt;SERVER-39353&lt;/del&gt;&lt;/a&gt; to enable chaining in the rollback test. Would you mind me repurposing &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39497&quot; title=&quot;Revert SERVER-33248&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39497&quot;&gt;&lt;del&gt;SERVER-39497&lt;/del&gt;&lt;/a&gt; to revert &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39353&quot; title=&quot;disable chaining in rollback_test.js&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39353&quot;&gt;&lt;del&gt;SERVER-39353&lt;/del&gt;&lt;/a&gt;? &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39497&quot; title=&quot;Revert SERVER-33248&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39497&quot;&gt;&lt;del&gt;SERVER-39497&lt;/del&gt;&lt;/a&gt; has been done as part of my patch anyway.&lt;/p&gt;</comment>
                            <comment id="2163844" author="xgen-internal-githook" created="Wed, 27 Feb 2019 01:52:24 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Siyuan Zhou&apos;, &apos;username&apos;: &apos;visualzhou&apos;, &apos;email&apos;: &apos;siyuan.zhou@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39367&quot; title=&quot;lastOpCommitted being reset on restart can cause sync source cycle&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39367&quot;&gt;&lt;del&gt;SERVER-39367&lt;/del&gt;&lt;/a&gt; Advance commit point when it has the same term as the last applied&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Reverted &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; &quot;Allow choosing a sync source that we are up to date&lt;br/&gt;
  with if it has a higher lastOpCommitted&quot;.&lt;/li&gt;
	&lt;li&gt;Reverted the logic change of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27123&quot; title=&quot;Only update commit point via spanning tree&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27123&quot;&gt;&lt;del&gt;SERVER-27123&lt;/del&gt;&lt;/a&gt; &quot;Only update commit point via&lt;br/&gt;
  spanning tree&quot;.&lt;/li&gt;
	&lt;li&gt;Added the term check when advancing commit point and added unit tests.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a78f546ff0caab8c33738aa922a294d9143a5397&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a78f546ff0caab8c33738aa922a294d9143a5397&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2163098" author="judah.schvimer" created="Tue, 26 Feb 2019 16:08:03 +0000"  >&lt;blockquote&gt;
&lt;p&gt;This problem and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; are both liveness problems and can be solved by restart or a replSetSyncFrom command&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Are you saying the &quot;minimal backport&quot; will not fix the sync source cycle problem completely?&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;take the minimum of commit point and last applied on learning via spanning tree&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;One side effect of this will be that it will be slightly slower to advance the commit point on secondaries since the commit point cannot be ahead of &lt;tt&gt;lastApplied&lt;/tt&gt; even if they&apos;re in the same term. This could cause some cache pressure, but I don&apos;t think a significant amount.&lt;/p&gt;</comment>
                            <comment id="2162291" author="tess.avitabile" created="Mon, 25 Feb 2019 22:07:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, yes, that solution sounds right to me.&lt;/p&gt;</comment>
                            <comment id="2162205" author="siyuan.zhou@10gen.com" created="Mon, 25 Feb 2019 21:14:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, yes. According to our discussion at the standup. We&apos;ll seek a minimal backport to 3.6 and (perhaps 3.4) by adding the term check. This problem and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; are both liveness problems and can be solved by restart or a &lt;tt&gt;replSetSyncFrom&lt;/tt&gt; command. A new write will resolve &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; and &lt;tt&gt;maxSyncSourceLagSecs&lt;/tt&gt; (default to 30secs) can break the sync source cycle.&lt;/p&gt;

&lt;p&gt;Mistaking the diverged branch as committed is more about a correctness issue though. The above proposal to take the minimum of commit point and last applied on learning via spanning tree seems to fix the correctness issue as well. We may be able to just back port this fix and leaving the liveness issues out for now.&lt;/p&gt;</comment>
                            <comment id="2162189" author="siyuan.zhou@10gen.com" created="Mon, 25 Feb 2019 20:54:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; has a concern that if a secondary is lagged, then it won&apos;t be able to update its commit point and throw away history.&lt;/p&gt;

&lt;p&gt;Actually, when a secondary is lagged, it can still learn of the latest commit point far ahead of it as long as the commit point&apos;s term is the same as its last applie&apos;s, so the gap should be the same as that on other up-to-date nodes, even if the stale node cannot update its commit point after the failover until it reaches the latest term. The only problem I can think of is after restarting the stale node, it forgot the commit point in the old term and isn&apos;t able to update its commit point. &lt;/p&gt;

&lt;p&gt;One solution is that we can relax the term check when learning from sync source but only update its commit point to &lt;tt&gt;min(commit point, my last applied)&lt;/tt&gt;. Given that requiring the term check on learning commit point ensures that the commit point is always on a node&apos;s branch, spanning tree ensures the syncing node is on the same branch as the sync source, so the syncing node knows it&apos;s on the same branch as the commit point even if they have different terms. CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, if this sounds right to you, I can file another ticket for it.&lt;/p&gt;</comment>
                            <comment id="2161351" author="judah.schvimer" created="Mon, 25 Feb 2019 13:48:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, does the above describe a bug in previous versions? If so, I think we need to fix that on 3.4 and 3.6 in addition to 4.0 and 4.2 which will likely get this sync source cycle fix. This seems like a bug we could easily miss since I think it would only occur doing majority reads on secondaries. Is that correct?&lt;/p&gt;</comment>
                            <comment id="2160862" author="siyuan.zhou@10gen.com" created="Sun, 24 Feb 2019 02:18:07 +0000"  >&lt;p&gt;The current way of learning commit point from spanning tree can still learn of a commit point in higher terms from its sync source, then switch to a stale branch and mark the stale branch as committed by mistake. Here&apos;s a concrete example with 5 nodes. Initially Node A is the primary in term 1. Node E is delayed.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;A: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;B: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;C: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E: [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Node B steps up in term 2, writes an entry.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;A: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;B: [1] [1] [2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;C: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D: [1] [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E: [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Node A steps up again in term 3 with votes from A, C and D and make its new oplog entry in term 3 majority committed.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;A: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;B: [1] [1] [2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;C: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E: [1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Node E syncs from A, and learned the new commit point in term 3.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;A: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;B: [1] [1] [2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;C: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E: [1] [1]           // It knows of the latest commit point in term 3&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Then Node E switches its sync source to B, replicates the stale branch of term 2 and mistakes that branch as committed.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;A: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;B: [1] [1] [2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;C: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D: [1] [1] [3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E: [1] [1] [2]   // It mistakes the entry in term 2 as committed.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This problem can be solved by only learning the commit point if it&apos;s in the same branch as mine, by comparing the terms of my lastApplied and the commit point.&lt;/p&gt;</comment>
                            <comment id="2141787" author="william.schultz" created="Thu, 7 Feb 2019 22:09:28 +0000"  >&lt;p&gt;I&apos;m on board with the heartbeat propagation solution proposed by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt; as well. If it&apos;s easier to implement than the alternative, it seems to safely achieve the desired goal.&lt;/p&gt;</comment>
                            <comment id="2141582" author="judah.schvimer" created="Thu, 7 Feb 2019 19:46:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, if we wanted to go with updating &lt;tt&gt;lastOpCommitted&lt;/tt&gt; based on a validated sync source candidate (we would still have to do some of the &lt;tt&gt;OplogFetcher::checkRemoteOplogStart&lt;/tt&gt; checks), I would do that on top of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; rather than reverting it, given the pains we had across the board before &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, I think your solution regarding heartbeats is safe and I really like it. It seems clean, easy, and accomplishes the same goal. I would propose potentially doing this in conjunction with a partial revert of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2140541" author="siyuan.zhou@10gen.com" created="Thu, 7 Feb 2019 00:10:30 +0000"  >&lt;p&gt;Will made a good point. In general, imagine two branches of the spanning tree sharing the same root (the primary), both branches can propagate their oplog entries and timestamps at their own paces. When the nodes from two branches check with each other, the problem in description becomes possible. Will gave a concrete example.&lt;/p&gt;

&lt;p&gt;The problem of propagation of commit point via heartbeat on secondaries is that the commit point can be from a diverged history. We force the commit point to flow through the spanning tree to make sure the commit point is on the same history branch. Alternatively, we could only update the commit point via heartbeat on secondaries &lt;b&gt;if the commit point&apos;s term is the same as my last applied&apos;s term&lt;/b&gt;. Thus we are guaranteed to be on the same branch with the commit point. We can revert &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; and have commit point propagate via heartbeats then.&lt;/p&gt;</comment>
                            <comment id="2138814" author="tess.avitabile" created="Tue, 5 Feb 2019 20:55:46 +0000"  >&lt;p&gt;I like the idea of allowing a node to update its lastOpCommitted based on a sync source candidate, even if it cannot select that sync source. I wonder if it&apos;s better to revert &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; first and then try this change, or try to do them at the same time, so that we don&apos;t affect tests. What do you think?&lt;/p&gt;</comment>
                            <comment id="2138184" author="judah.schvimer" created="Tue, 5 Feb 2019 16:06:14 +0000"  >&lt;p&gt;I think reverting &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; is probably the best way forward, though not having &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; caused many headaches, so I want to consider alternatives first.&lt;/p&gt;

&lt;p&gt;I&apos;ve been thinking about two ideas, though:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Can &lt;tt&gt;OplogFetcher:checkRemoteOplogStart&lt;/tt&gt; say that the sync source is invalid if the &lt;tt&gt;lastApplied&lt;/tt&gt; optimes are equal but the &lt;tt&gt;lastOpCommitted&lt;/tt&gt; optimes are not, but still allow the &lt;tt&gt;OplogFetcher&lt;/tt&gt; to process the metadata it receives? This could allow us to say a sync source is invalid (preventing a cycle), but still advance our &lt;tt&gt;lastOpCommitted&lt;/tt&gt; safely (since we&apos;ve acknowledged that in this state it is safe to trust the sync source&apos;s metadata, it just may cause a cycle).&lt;/li&gt;
	&lt;li&gt;When chaining is disabled, we attempt to sync from the primary no matter what and then &lt;tt&gt;OplogFetcher:checkRemoteOplogStart&lt;/tt&gt; says it is invalid if it is not ahead. If there are no valid sync sources, and we implemented (1) above, we could choose our knowledge of the primary as our sync source candidate, and only use it for its metadata. I do not think it would be safe to sync from nodes we think are the primary if they are only ahead of us by &lt;tt&gt;lastOpCommitted&lt;/tt&gt;, since there could still be a cycle of outdated information about the current primary. However, there is little harm in sending an extra &lt;tt&gt;find&lt;/tt&gt; command to a node if the alternative is doing nothing. I&apos;m also, however, not 100% sure that this solves the problems &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;. Are there circumstances where the node we think is primary does not have the most up-to-date &lt;tt&gt;lastOpCommitted&lt;/tt&gt;? I think that if the cluster is healthy (which is the state &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; is trying to solve), this is a fair assumption, at least eventually (which is all that matters for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;It seems &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; concurrently had a similar idea.&lt;/p&gt;</comment>
                            <comment id="2138166" author="william.schultz" created="Tue, 5 Feb 2019 15:59:09 +0000"  >&lt;p&gt;One other thought: For two nodes A, B where:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;A.lastApplied = B.lastApplied&lt;/li&gt;
	&lt;li&gt;A.lastCommitted &amp;lt; B.lastCommitted&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;we could consider allowing node A to update its lastCommittedOpTime from B &lt;em&gt;without&lt;/em&gt; choosing it as a sync source. It should be safe for A to learn of a new commit point from a node that has the same oplog as itself, but we wouldn&apos;t allow A to actually start syncing from B. This may address the original issue raised in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt; about secondary majority reads not progressing, but avoid the sync source cycle formation scenario outlined above.&lt;/p&gt;</comment>
                            <comment id="2137994" author="william.schultz" created="Tue, 5 Feb 2019 14:36:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; Sorry, there was a typo in step 7 of my previous comment. The full sequence with step 7 fixed is instead:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;All nodes initially have lastApplied=1, lastCommitted=1. n0 is primary.&lt;/li&gt;
	&lt;li&gt;n0 applies operations at timestamps 2, 3, and 4 updates it lastApplied to 4.&lt;/li&gt;
	&lt;li&gt;n0 replicates all of its operations to n1, updating n1.lastApplied=4.&lt;/li&gt;
	&lt;li&gt;n1 gets partitioned from all other nodes.&lt;/li&gt;
	&lt;li&gt;n0 replicates its operation at timestamp 2 to nodes n2 and n3 and it becomes majority committed.&lt;/li&gt;
	&lt;li&gt;n2 and n3 replicate the operation at timestamp 3 from the primary and learn of the new commit point (timestamp 2) at the same time. At this point we have n0.lastCommitted=2, n2.lastCommitted=2 and n3.lastCommitted=2.&lt;/li&gt;
	&lt;li&gt;We now have n2.lastApplied=3, n2.lastCommitted=2 and n1.lastApplied=4, n1.lastCommitted=1.&lt;/li&gt;
	&lt;li&gt;n2 chooses to sync from n1 because n1.lastApplied=4 &amp;gt; n2.lastApplied=3&lt;/li&gt;
	&lt;li&gt;n2 catches up with n1 so that n2.lastApplied=n1.lastApplied=4. We still have n2.lastCommitted=2&lt;/li&gt;
	&lt;li&gt;n1 chooses to sync from n2 because n2.lastCommitted=2 &amp;gt; n1.lastCommitted=1.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I wrote &quot;n0&quot; instead of &quot;n1&quot; in step 7 in the original comment. Step 7 is what sets the stage for a &lt;em&gt;possible&lt;/em&gt; cycle. When I said &quot;pre-condition&quot;, I meant that it was a state that &lt;em&gt;could&lt;/em&gt; allow for a cycle to subsequently form based solely on the sync source selection rules, even though it might not. The last 3 steps I added produce the actual cycle. It looks like &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; already corrected my errors and demonstrated an analogous cycle case above.&lt;/p&gt;</comment>
                            <comment id="2137967" author="tess.avitabile" created="Tue, 5 Feb 2019 14:19:12 +0000"  >&lt;p&gt;Once we have state 7, we can get to a sync source cycle:&lt;/p&gt;

&lt;p&gt;8. n2 syncs from n0 because n2.lastApplied=3&amp;lt;4=n0.lastApplied.&lt;/p&gt;

&lt;p&gt;9. n2 catches up to n0, so n2.lastApplied=4=n0.lastApplied.&lt;/p&gt;

&lt;p&gt;10. n0 syncs from n2 because n0.lastApplied=4=n2.lastApplied and n0.lastCommitted=1&amp;lt;2=n2.lastCommitted.&lt;/p&gt;

&lt;p&gt;I think we may need to revert&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33248&quot; title=&quot;Allow choosing a sync source that we are up to date with if it has a higher lastOpCommitted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33248&quot;&gt;&lt;del&gt;SERVER-33248&lt;/del&gt;&lt;/a&gt;. I think the fallacy of that work was the assumption that if A is syncing from B, then A.lastCommitted&amp;lt;=B.lastCommitted. Another option is to add the requirement for sync source selection that source.lastCommitted&amp;gt;=self.lastCommitted. However, this may have unintended consequences.&lt;/p&gt;</comment>
                            <comment id="2137941" author="judah.schvimer" created="Tue, 5 Feb 2019 13:48:01 +0000"  >&lt;blockquote&gt;
&lt;p&gt;n2 and n3 replicate the operation at timestamp 3 from the primary and learn of the new commit point (timestamp 2) at the same time. At this point we have n0.lastCommitted=2, n2.lastCommitted=2 and n3.lastCommitted=2.&lt;br/&gt;
We now have n2.lastApplied=3, n2.lastCommited=2 and n0.lastApplied=4, n0.lastCommitted=1.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt;, these two lines seem to disagree on the state of n0. &lt;/p&gt;

&lt;p&gt;Also, just being at state 7 does not mean we&apos;ll develop a cycle. At step 7, the node with the higher last applied should not choose the other node as a sync source because it has a lower lastApplied, even though its lastCommitted is greater.&lt;/p&gt;</comment>
                            <comment id="2137695" author="william.schultz" created="Tue, 5 Feb 2019 04:02:21 +0000"  >&lt;p&gt;I do think that it is possible in general for two replica set nodes, A, B to satisfy:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;A.lastApplied &amp;gt; B.lastApplied&lt;/li&gt;
	&lt;li&gt;A.lastCommitted &amp;lt; B.lastCommitted&lt;/li&gt;
	&lt;li&gt;A.lastCommitted != 0&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The first two bullets seem to be the key pre-condition for sync source cycle formation. Consider the following scenario in a 4 node replica set with nodes n0, n1, n2, n3:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;All nodes initially have lastApplied=1, lastCommitted=1. n0 is primary.&lt;/li&gt;
	&lt;li&gt;n0 applies operations at timestamps 2, 3, and 4 updates it lastApplied to 4.&lt;/li&gt;
	&lt;li&gt;n0 replicates all of its operations to n1, updating n1.lastApplied=4.&lt;/li&gt;
	&lt;li&gt;n1 gets partitioned from all other nodes.&lt;/li&gt;
	&lt;li&gt;n0 replicates its operation at timestamp 2 to nodes n2 and n3 and it becomes majority committed.&lt;/li&gt;
	&lt;li&gt;n2 and n3 replicate the operation at timestamp 3 from the primary and learn of the new commit point (timestamp 2) at the same time. At this point we have n0.lastCommitted=2, n2.lastCommitted=2 and n3.lastCommitted=2.&lt;/li&gt;
	&lt;li&gt;We now have n2.lastApplied=3, n2.lastCommited=2 and n0.lastApplied=4, n0.lastCommitted=1.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;This establishes the pre-condition referenced above, and should be sufficient for a sync source cycle to subsequently form. If n2 and n1 were to be re-connected directly to each other, they would be able to form a 2 node cycle. This repro (&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/208066/208066_sync_source_cycle.js&quot; title=&quot;sync_source_cycle.js attached to SERVER-39367&quot;&gt;sync_source_cycle.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;) demonstrates a roughly analogous case.&lt;/p&gt;</comment>
                            <comment id="2137095" author="judah.schvimer" created="Mon, 4 Feb 2019 19:10:28 +0000"  >&lt;p&gt;This fix should revert &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39353&quot; title=&quot;disable chaining in rollback_test.js&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39353&quot;&gt;&lt;del&gt;SERVER-39353&lt;/del&gt;&lt;/a&gt; (18dde98af61a71b1bc5ec990c2184c5725d9b0c0) to re-enable chaining in &lt;tt&gt;rollback_test.js&lt;/tt&gt;.&lt;/p&gt;</comment>
                            <comment id="2137090" author="judah.schvimer" created="Mon, 4 Feb 2019 19:08:45 +0000"  >&lt;p&gt;A key question that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; raised is: &quot;is it impossible for two nodes A, B to have &lt;tt&gt;A.lastApplied &amp;gt; B.lastApplied but A.lastCommitted &amp;lt; B.lastCommitted?&lt;/tt&gt;&quot;, and if it&apos;s possible, is it only possible when &lt;tt&gt;A.lastCommitted = 0&lt;/tt&gt;?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="687742">SERVER-39497</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="718638">SERVER-40193</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="718640">SERVER-40194</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="333019">SERVER-27123</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="682306">SERVER-39353</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="687742">SERVER-39497</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="698913">SERVER-39626</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="704390">SERVER-39831</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="495970">SERVER-33248</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="208066" name="sync_source_cycle.js" size="4181" author="william.schultz@mongodb.com" created="Tue, 5 Feb 2019 04:03:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>24.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 5 Feb 2019 04:02:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 47 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1335</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 47 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hul6c7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hub17j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2822">Repl 2019-03-11</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hukslj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>