[SERVER-10793] Start election if more than one primary is online Created: 16/Sep/13  Updated: 11/Jul/16  Resolved: 15/Jan/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.4.10, 2.5.5

Type: Improvement Priority: Major - P3
Reporter: Scott Hernandez (Inactive) Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-10768 add proper support for SIGSTOP and SI... Closed
Participants:

 Description   
Issue Status as of March 28, 2014

ISSUE SUMMARY
Replica sets should only ever contain at most one primary node. If a primary detects another primary in the replica set via the heartbeat messages, the current behavior would force the primary to step down only if its _id in the replica set configuration is higher than the other primary's _id. The intention of this was to only step down one of the primaries, thus avoiding a new election. However, since the _id is chosen arbitrarily and does not indicate priority, this can lead to a lower-priority member remaining as the primary node. Another issue is a one-way network partition, which could potentially lead to multiple primary nodes for prolonged times.

USER IMPACT
This bug can lead to a primary node that does not have the highest priority, or in rare cases (i.e. with transient network issues) to multiple primaries for prolonged times. The latter situation can affect data integrity.

SOLUTION
The fix is to unconditionally step down all primary nodes if multiple primary nodes are detected. While this can cause elections in more cases than before, it is safer than having the wrong primary, or potentially multiple primaries.

WORKAROUNDS
In situations where a lower-priority node remains the primary, a forced election with rs.stepDown() can promote the higher-priority node back to primary.

AFFECTED VERSIONS
All versions from 2.2.0 to 2.4.9 are affected.

PATCHES
The fix is included in the 2.4.10 production release and the 2.5.5 development release, which will evolve into the 2.6.0 production release.

Original Description

Check at every heartbeat, as it comes in, that the state of the world shows only one primary at most. If more than one is found, start an election.



 Comments   
Comment by Githook User [ 09/Mar/14 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-10793 always step down when another primary is seen
Branch: v2.4
https://github.com/mongodb/mongo/commit/a29de43de7fee23456fe44fcd4e00606427d2997

Comment by Eric Milkie [ 16/Jan/14 ]

That is correct. The other ticket is scheduled to be fixed with the larger refactor of election protocol.

Comment by Zardosht Kasheff [ 16/Jan/14 ]

I think this solution only works if the primaries see each other. If you have a network issue where the two primaries don't see each other, you can have two indefinite primaries. See https://jira.mongodb.org/browse/SERVER-9848.

Comment by Githook User [ 15/Jan/14 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-10793 always step down when another primary is seen
Branch: master
https://github.com/mongodb/mongo/commit/bebbcac06efd48abb2f9224150c7ae93b85fcf72

Comment by Eric Milkie [ 18/Oct/13 ]

Ok so if I'm a primary node and I receive a heartbeat that someone else is primary, I should step down (at which point, the election logic will take over and hopefully end up with just 1 primary in the end).
We already have code to do that, it's manager.cpp:95. We try to make sure only one primary steps down, but this logic is flawed if visibility in the cluster is not reflexive.

Comment by Scott Hernandez (Inactive) [ 18/Oct/13 ]

I was thinking that the primaries would take the action when they got the heartbeat. This was to try to cover the case that they got each others heartbeats (each showing that they were primary), not that a 3rd party getting the heartbeats.

In the case it is a 3rd party we could use a new command (to inform both) but it seems like the only time we would need this would be when the two primaries can't directly communicate, since they should be able to react directly to heartbeats otherwise.

Comment by Eric Milkie [ 18/Oct/13 ]

It's not going to be that simple. At each heartbeat, it's possible that you simply see one node step up to be primary before you get the heartbeat from the node that stepped down.
When you see more than one primary, what should your action be? Run stepdown on the primaries you see? Or simply attempt to elect yourself? I would imagine as soon as you did that, both primaries would veto your election attempt.

Generated at Thu Feb 08 03:24:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.