<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:13:52 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-47331] Rethink the transition from force reconfig to safe reconfig</title>
                <link>https://jira.mongodb.org/browse/SERVER-47331</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When the current config C0 is installed by a &quot;force&quot; reconfig, the next non-force reconfig with config C1 doesn&apos;t prevent config divergence if&lt;br/&gt;
1. Reconfig C1 has not propagated to a majority of nodes.&lt;br/&gt;
2. A failover happens&lt;br/&gt;
3. A new reconfig with a different config C2 runs on the new primary.&lt;br/&gt;
4. C1 and C2 propagate to disjoint nodes.&lt;/p&gt;

&lt;p&gt;The diverged configs may lead to two primaries elected in the same term until C2 (with a higher config term) propagates to a majority of C1. A similar issue is shown in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47119&quot; title=&quot;Config term does not get initialized until replSetReconfig is run&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47119&quot;&gt;&lt;del&gt;SERVER-47119&lt;/del&gt;&lt;/a&gt; with a detailed trace.&lt;/p&gt;

&lt;p&gt;In Initial Sync Semantics project, we will give new nodes votes: 0 and run automatic reconfig afterwards to grant them votes afterwards. The config to add the node will face the unsafe but rare case mentioned above. Once the first reconfig passes the aforementioned unsafe period and becomes committed, the following automatic reconfigs will be safe.&lt;/p&gt;

&lt;p&gt;To avoid the unsafe case, one idea is to run an automatic reconfig after a force reconfig by increasing the config version and giving it a config term. After this automatic reconfig, following reconfigs will be safe. However, when users run &quot;force&quot; reconfig, it&apos;s likely the replset is not stable so that they are willing to risk the loss of committed data. It may not be the right time to run such an automatic reconfig.&lt;/p&gt;

&lt;p&gt;Even worse, the automatic reconfig may interrupt the propagation of the &quot;force&quot; reconfig. For example, assuming the current config C0 has 5 nodes, a force reconfig C1 runs on a secondary to convert that secondary to a single node replica set. The force reconfig C1 will increase the version but remove the config term, then propagate to other nodes on their next heartbeats. Nodes in C0 will become REMOVED after learning C1. However, if an automatic reconfig C2 happens on the single node replset, since C2 has a term, C2&apos;s term has to be higher than C0 to propagate, which may not be the case if another election occurs in C0. As a result, C2 may not be able to propagate to nodes still in C0. If their terms are the same, nodes in C0 will have a diverged config. They&apos;ll be alive and keep running heartbeats to the single node replset. When either of C0 or C2 has a higher term, it will be propagated to the other, potentially overriding the force reconfig.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1303792">SERVER-47331</key>
            <summary>Rethink the transition from force reconfig to safe reconfig</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="siyuan.zhou@mongodb.com">Siyuan Zhou</assignee>
                                    <reporter username="siyuan.zhou@mongodb.com">Siyuan Zhou</reporter>
                        <labels>
                    </labels>
                <created>Fri, 3 Apr 2020 18:58:21 +0000</created>
                <updated>Sun, 29 Oct 2023 22:09:56 +0000</updated>
                            <resolved>Mon, 13 Apr 2020 23:07:14 +0000</resolved>
                                                    <fixVersion>4.7.0</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="3037961" author="siyuan.zhou@10gen.com" created="Mon, 13 Apr 2020 23:07:14 +0000"  >&lt;p&gt;I wanted to mark this &quot;Done&quot;, but I have to go with &quot;Fixed&quot; to enable the downstream attention.&lt;/p&gt;</comment>
                            <comment id="3037958" author="siyuan.zhou@10gen.com" created="Mon, 13 Apr 2020 23:05:24 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, closing this.&lt;/p&gt;</comment>
                            <comment id="3036620" author="judah.schvimer" created="Mon, 13 Apr 2020 12:50:12 +0000"  >&lt;p&gt;I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47495&quot; title=&quot;Ban force reconfig with &amp;quot;newlyAdded&amp;quot; fields&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47495&quot;&gt;&lt;del&gt;SERVER-47495&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3036602" author="tess.avitabile" created="Mon, 13 Apr 2020 12:34:31 +0000"  >&lt;p&gt;Yes, that sounds good to me.&lt;/p&gt;</comment>
                            <comment id="3034241" author="judah.schvimer" created="Thu, 9 Apr 2020 21:38:45 +0000"  >&lt;p&gt;Thanks for the summary. I will file the ISS tickets once we agree on the above.&lt;/p&gt;</comment>
                            <comment id="3034152" author="siyuan.zhou@10gen.com" created="Thu, 9 Apr 2020 21:18:10 +0000"  >&lt;p&gt;Discussed with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=evin.roesle&quot; class=&quot;user-hover&quot; rel=&quot;evin.roesle&quot;&gt;evin.roesle&lt;/a&gt; in person. Since automatic reconfig in ISS runs on top of the first user-initiated reconfig command, their safety is guaranteed if the user-initiated reconfig is a safe reconfig. If the user-initiated reconfig is a force reconfig, then we won&apos;t add newlyAdded fields nor run automatic reconfig at all.&lt;/p&gt;

&lt;p&gt;The only edge case is when the the user-initiated reconfig is a force reconfig with &quot;newlyAdded&quot; fields. It will trigger automatic reconfig which will run on an unsafe config.&lt;/p&gt;

&lt;p&gt;There are a few options to solve this issue.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Ban force reconfig with &quot;newlyAdded&quot; fields&lt;/li&gt;
	&lt;li&gt;Allow force reconfig with &quot;newlyAdded&quot; fields assuming the following automatic reconfig will be safe in most cases.&lt;/li&gt;
	&lt;li&gt;Make safe reconfig after force reconfig safer. This will need a significant design that evaluates when it&apos;s safe to convert the force reconfig.&lt;/li&gt;
	&lt;li&gt;Leave the force config with &quot;newlyAdded&quot; fields as-is without running automatic reconfig. This leaves an incomplete state.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;We agreed to go with option 1 since &quot;newlyAdded&quot; is an internal field anyway. Beyond the behavioral change, we need to document that the transition from force reconfig to safe reconfig isn&apos;t safe. I&apos;m adding downstream change in this ticket. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, does the plan sound good to you?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, do you mind filing the corresponding ticket in ISS?&lt;/p&gt;</comment>
                            <comment id="3032019" author="siyuan.zhou@10gen.com" created="Wed, 8 Apr 2020 21:17:44 +0000"  >&lt;blockquote&gt;&lt;p&gt;If C0 can override the force reconfig, then that seems like a problem even if we don&apos;t do an automatic noop reconfig C2.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;C0 shouldn&apos;t override the force reconfig. The force reconfig should take effect immediately by having a much higher config number.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;If this noop automatic reconfig is particularly dangerous, then shouldn&apos;t any automatic reconfig be particularly dangerous? If that&apos;s the case, then we have to reconsider the ISS project&apos;s automatic reconfigs.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I don&apos;t think noop automatic reconfig in ISS is dangerous since the reconfig is initiated by a user. After a force reconfig, the first user-initiated safe reconfig to add a node is subject to all the potential issues of force reconfig. Its safety depends on the user as in other cases around force reconfig. In most cases, users would only run reconfig when the system is stable. The following ISS automatic reconfigs will then become safe.&lt;/p&gt;

&lt;p&gt;As you mentioned,&#160;automatic reconfig won&apos;t be safe after force reconfig with &quot;newlyAdded&quot;.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;If the force reconfig specifies &quot;newlyAdded&quot;, then once the primary sees that node is a secondary, the primary will initiate an automatic reconfig to remove &quot;newlyAdded&quot;.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I&apos;d suggest banning &quot;newlyAdded&quot; on force reconfig, since &quot;newlyAdded&quot; is supposed to be an internal field and force reconfig is supposed to only used in emergency.&lt;/p&gt;</comment>
                            <comment id="3031125" author="judah.schvimer" created="Wed, 8 Apr 2020 15:23:48 +0000"  >&lt;p&gt;I don&apos;t follow the final paragraph above.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;If C0 can override the force reconfig, then that seems like a problem even if we don&apos;t do an automatic noop reconfig C2.&lt;/li&gt;
	&lt;li&gt;If C2 overrides the force reconfig, isn&apos;t that fine since that&apos;s what we want in the first place? C2 was created with C1 as the &quot;base config&quot; so it&apos;s just a safe version of C1.&lt;/li&gt;
	&lt;li&gt;If this noop automatic reconfig is particularly dangerous, then shouldn&apos;t any automatic reconfig be particularly dangerous? If that&apos;s the case, then we have to reconsider the ISS project&apos;s automatic reconfigs.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I think that doing an automatic reconfig at the next chance we get would be good to narrow the window where the next reconfig will be unsafe, and could allow us to do other automatic reconfigs safely.&lt;/p&gt;</comment>
                            <comment id="3025813" author="judah.schvimer" created="Fri, 3 Apr 2020 19:11:32 +0000"  >&lt;p&gt;This behavior is implemented and tested in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46347&quot; title=&quot;Ensure that force reconfigs are not given a &amp;#39;newlyAdded&amp;#39; field&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46347&quot;&gt;&lt;del&gt;SERVER-46347&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46350&quot; title=&quot;Test that force reconfig removes a &amp;#39;newlyAdded&amp;#39; field if one exists but is not provided in the reconfig&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46350&quot;&gt;&lt;del&gt;SERVER-46350&lt;/del&gt;&lt;/a&gt;, and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46348&quot; title=&quot;Test behavior when a user provides a &amp;#39;newlyAdded&amp;#39; field in a force-reconfig&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46348&quot;&gt;&lt;del&gt;SERVER-46348&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3025808" author="judah.schvimer" created="Fri, 3 Apr 2020 19:10:09 +0000"  >&lt;p&gt;In ISS, a force reconfig will replace the current config verbatim and we will not rewrite it at all. Thus if the force reconfig does not specify &quot;newlyAdded&quot; that would remove the &quot;newlyAdded&quot; field from an existing node (if that node currently had &quot;newlyAdded&quot; specified). If the force reconfig specifies &quot;newlyAdded&quot;, then once the primary sees that node is a secondary, the primary will initiate an automatic reconfig to remove &quot;newlyAdded&quot;.&lt;/p&gt;</comment>
                            <comment id="3025794" author="siyuan.zhou@10gen.com" created="Fri, 3 Apr 2020 19:01:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, what&apos;s the current design of Initial Sync Semantics if the current config is from a force reconfig? I don&apos;t see any problem in terms of the automatic reconfig.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="1310099">DOCS-13582</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1312826">SERVER-47495</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 3 Apr 2020 19:10:09 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 43 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Transition from force reconfig to safe reconfig:&lt;br/&gt;
The safety of a new non-force reconfig is not guaranteed until the current config is installed by a non-force reconfig and committed.&lt;br/&gt;
&lt;br/&gt;
After a force reconfig and when the system becomes stable, running a new non-force reconfig with the same config content is recommended. When it becomes committed, future reconfigs will be guaranteed safe.</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 43 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxdeov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx10d3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3769">Repl 2020-04-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxd0y7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>