[SERVER-9730] Asymmetrical network partition can cause the election of two PRIMARY nodes Created: 20/May/13  Updated: 11/Jul/16  Resolved: 22/May/13

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.2.4, 2.4.3, 2.5.0
Fix Version/s: 2.2.5, 2.4.4, 2.5.1

Type: Bug Priority: Critical - P2
Reporter: William Zola Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-8145 Two primaries for the same replica set Closed
Related
related to SERVER-9283 lastHeartbeatRecv flaps between inval... Closed
is related to SERVER-9756 network partition where minority was ... Closed
Operating System: ALL
Participants:

 Description   

Consider a replica set with three nodes: A, B, C. 'A' is PRIMARY, 'B' and 'C' are SECONDARY.

If there is a network partition between 'A' and 'C', then:

  • A is still PRIMARY because it can see a majority of the set
  • C will start an election trying to become PRIMARY
    • It can see a majority of the set
    • It cannot see a primary

If, for whatever reason, 'B' allows 'C' to become PRIMARY, then the set will have two PRIMARY nodes until one of them is stepped down.



 Comments   
Comment by Volodymyr Gren [ 18/Jul/13 ]

thanks, now it's super clear

Comment by William Zola [ 18/Jul/13 ]

abramovich@nimble.com – You're correct: I was mistaken.

Comment by Volodymyr Gren [ 17/Jul/13 ]

So looks William was not right in his statement https://jira.mongodb.org/browse/SERVER-9730?focusedCommentId=383129&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-383129

Comment by Eric Milkie [ 17/Jul/13 ]

An environment where there is an arbiter, primary, and secondary is not affected by this bug.

Comment by Volodymyr Gren [ 17/Jul/13 ]

So if I correctly understood Eric - we cant face this bug in environment when there is arbiter, primary and secondary, right?

just environment with primary and two secondaries could be affected?

Comment by Eric Milkie [ 17/Jul/13 ]

No, because the secondary will never be perceived to be fresher than the primary by the arbiter, since the secondary cannot chain through the arbiter to reach the primary's replicated ops. Arbiters do not do anything other than participate in elections; notably, they do not forward along replicated ops, as the B Secondary in the original scenario was doing.

Comment by Volodymyr Gren [ 17/Jul/13 ]

Let's say without B and C

I have 1 primary, 1 secondary, 1 arbiter; can I face this bug so I'll have two primaries and 1 arbiter?

looks yes - when primary will stop see the secondary and vice versa, but arbiter will see both (and vice versa) - the secondary will also become master

so the questions are:

1) what will happen to mongoses which are seeing two primaries in a shard
2) what will happen after connectivity is back?

thanks

Comment by William Zola [ 17/Jul/13 ]

> yes, but can the B become PRIMARY - so we'll have A and B as primaries and C as arbiter

I think the question you meant to ask was: "Can this scenario occur if 'B' is an arbiter?". The answer to that question is "yes".

Comment by Eric Milkie [ 17/Jul/13 ]

No, B will not become PRIMARY because it can still see A is PRIMARY in the original scenario.

Comment by Volodymyr Gren [ 17/Jul/13 ]

yes, but can the B become PRIMARY - so we'll have A and B as primaries and C as arbiter

Comment by William Zola [ 28/May/13 ]

can this happen when 'C' in Description scenario is an arbiter?

No it cannot. If 'C' is an arbiter, it will not attempt to become primary.

Comment by Volodymyr Gren [ 27/May/13 ]

can this happen when 'C' in Description scenario is an arbiter?

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-9730 under any circumstances, do not vote yea while a primary already exists
Branch: v2.2
https://github.com/mongodb/mongo/commit/5ec76b730a94b6d8d072fab0a348e34debe72ea3

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-9730 under any circumstances, do not vote yea while a primary already exists
Branch: v2.4
https://github.com/mongodb/mongo/commit/d01732d65dea9295ffe08bbbe748136dcd4178bf

Comment by auto [ 22/May/13 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-9730 under any circumstances, do not vote yea while a primary already exists
Branch: master
https://github.com/mongodb/mongo/commit/47fc9f6809a29713ac7d80a4b318e690906075df

Comment by Scott Hernandez (Inactive) [ 20/May/13 ]

Can you repro this, pref. with a jstest?

You can use use the bridging and partitioning in ReplSetTest to create many scenarios, like the one described. See https://github.com/mongodb/mongo/blob/master/jstests/replsets/no_chaining.js for an example.

Generated at Thu Feb 08 03:21:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.