<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:00 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26717] PSA flapping during netsplit when using PV1</title>
                <link>https://jira.mongodb.org/browse/SERVER-26717</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Under PV1 when using a PSA (or PSSSA) replset spread across three data centres, the primary node flaps between DC1 and DC2 every 10 seconds during a netsplit between DC1 and DC2. Each data centre receives roughly half the writes (assuming roughly constant write traffic). When the netsplit is resolved, the data in the non-primary data centre is rolled back.&lt;/p&gt;

&lt;p&gt;When the netsplit occurs, the following sequence of events happen:&lt;br/&gt;
1. Secondary in DC2 is unable to contact a primary for 10 seconds and calls a new term.&lt;br/&gt;
2. The DC3 arbiter announces the new term to DC1.&lt;br/&gt;
3. The DC1 primary steps down.&lt;br/&gt;
4. Client connections are dropped.&lt;br/&gt;
5. The node in DC2 is elected primary.&lt;br/&gt;
6. Clients reconnect and find DC2 is now primary. DC2 starts accepting writes.&lt;br/&gt;
7. 10 seconds later, DC1 hasn&#8217;t been able to contact a primary and the process repeats itself.&lt;/p&gt;

&lt;p&gt;Here is a snippet of logs from the arbiter demonstrating the flapping behaviour:&lt;br/&gt;
2016-10-19T22:49:47.655+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.102:27018 is now in state SECONDARY&lt;br/&gt;
2016-10-19T22:49:47.669+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.101:27017 is now in state PRIMARY&lt;br/&gt;
2016-10-19T22:49:57.672+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.102:27017 is now in state PRIMARY&lt;br/&gt;
2016-10-19T22:50:02.672+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; dropping unhealthy pooled connection to 10.0.0.101:27017&lt;br/&gt;
2016-10-19T22:50:02.672+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; after drop, pool was empty, going to spawn some connections&lt;br/&gt;
2016-10-19T22:50:02.673+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Connecting to 10.0.0.101:27017&lt;br/&gt;
2016-10-19T22:50:02.674+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Successfully connected to 10.0.0.101:27017&lt;br/&gt;
2016-10-19T22:50:02.675+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.101:27017 is now in state SECONDARY&lt;br/&gt;
2016-10-19T22:50:12.676+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; dropping unhealthy pooled connection to 10.0.0.102:27017&lt;br/&gt;
2016-10-19T22:50:12.676+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; after drop, pool was empty, going to spawn some connections&lt;br/&gt;
2016-10-19T22:50:12.676+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Connecting to 10.0.0.102:27017&lt;br/&gt;
2016-10-19T22:50:12.677+0000 I ASIO     &lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Successfully connected to 10.0.0.102:27017&lt;br/&gt;
2016-10-19T22:50:12.677+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.101:27018 is now in state PRIMARY&lt;br/&gt;
2016-10-19T22:50:12.678+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.102:27017 is now in state SECONDARY&lt;br/&gt;
2016-10-19T22:50:22.665+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.0.102:27018 is now in state PRIMARY&lt;/p&gt;

&lt;p&gt;N.B. Flapping does not occur with PSS/PV1 or PSA/PV0.&lt;/p&gt;</description>
                <environment></environment>
        <key id="325758">SERVER-26717</key>
            <summary>PSA flapping during netsplit when using PV1</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="james.kovacs@mongodb.com">James Kovacs</reporter>
                        <labels>
                    </labels>
                <created>Thu, 20 Oct 2016 22:12:53 +0000</created>
                <updated>Tue, 6 Dec 2022 04:13:43 +0000</updated>
                            <resolved>Wed, 18 Jan 2017 19:24:03 +0000</resolved>
                                    <version>3.2.10</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="1479614" author="spencer" created="Wed, 18 Jan 2017 19:23:48 +0000"  >&lt;p&gt;The plan to address this is no longer to implement &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14539&quot; title=&quot;Full consensus arbiter (i.e. uses an oplog)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14539&quot;&gt;SERVER-14539&lt;/a&gt;.  Instead we are going to do the smaller change of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27125&quot; title=&quot;Arbiters in pv1 should vote no in elections if they can see a healthy primary of equal or greater priority to the candidate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27125&quot;&gt;&lt;del&gt;SERVER-27125&lt;/del&gt;&lt;/a&gt;, which is much less invasive and can be completed much sooner.  I&apos;m closing this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27125&quot; title=&quot;Arbiters in pv1 should vote no in elections if they can see a healthy primary of equal or greater priority to the candidate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27125&quot;&gt;&lt;del&gt;SERVER-27125&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1428283" author="spencer" created="Mon, 7 Nov 2016 18:36:35 +0000"  >&lt;p&gt;Since we didn&apos;t wind up doing &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26725&quot; title=&quot;Automatically reconfig pv1 replica sets using priorities or arbiters to pv0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26725&quot;&gt;&lt;del&gt;SERVER-26725&lt;/del&gt;&lt;/a&gt;, this issue still exists with the use of arbiters in pv1.  &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14539&quot; title=&quot;Full consensus arbiter (i.e. uses an oplog)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14539&quot;&gt;SERVER-14539&lt;/a&gt; remains the long term plan to improve this behavior.&lt;/p&gt;</comment>
                            <comment id="1414761" author="spencer" created="Fri, 21 Oct 2016 19:08:15 +0000"  >&lt;p&gt;Assuming current primary is in DC1...&lt;/p&gt;

&lt;p&gt;This works in PV1 with PSS because in that case the secondary in DC2 would vote no to the node in DC3 becoming primary because it would be ahead of that in terms of replication from writes that DC1 took.&lt;br/&gt;
This works in PV0 with PSA because in that case the arbiter in in DC2 would vote no to the node in DC3 becoming primary because it can still see a healthy primary.&lt;/p&gt;

&lt;p&gt;This will be fixed short term by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26725&quot; title=&quot;Automatically reconfig pv1 replica sets using priorities or arbiters to pv0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26725&quot;&gt;&lt;del&gt;SERVER-26725&lt;/del&gt;&lt;/a&gt; and longer term by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14539&quot; title=&quot;Full consensus arbiter (i.e. uses an oplog)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14539&quot;&gt;SERVER-14539&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="333037">SERVER-27125</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="146793">SERVER-14539</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="325959">SERVER-26728</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="325918">SERVER-26725</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000VWsHjIAL]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 21 Oct 2016 19:08:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 4 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-722</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 4 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>james.kovacs@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjsxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsr2bj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;1. Create a 3-member PSA replset spread across 3 VMs, which will represent our 3 DCs. DC1 and DC2 contain databearing nodes and DC3 contains the arbiter.&lt;br/&gt;
2. Create a netsplit between DC1 and DC2. e.g. `sudo iptables -I INPUT -s OTHER_VM_IP -j DROP` on each of DC1 and DC2&lt;br/&gt;
3. View `mongod.log` on DC3 to watch primary flap between DC1 and DC2 every 10 seconds. This can also be seen via `rs.status()`.&lt;br/&gt;
4. Perform writes to the replset long enough so that some writes go to DC1 and others go to DC2. (e.g. &amp;gt;10 seconds)&lt;br/&gt;
5. Resolve netsplit. e.g. `sudo iptables -R INPUT 1` on DC1 and DC2&lt;br/&gt;
6. Either DC1 or DC2 will go into a ROLLBACK state and its writes dumped to disk.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsefcf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>