[SERVER-18889] Reelection waiting for a dead system to vote Created: 09/Jun/15  Updated: 09/Jun/15  Resolved: 09/Jun/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Alex Kotenko Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

tl;dr
Former Primary becomes Secondary and fails reelection waiting indefinitely for dead node to vote when another Secondary node is cleanly shut down, while third node is known to be dead. This renders whole replica set unresponsive while there is a healthy server - one supposed to be the Primary.

Whole thing
The system I'm running consists of three servers in a replica set, two active user facing nodes (mongo01 & mongo02) and one hidden node (dbslave01), for backups. All three are voting, mongo01 & mongo02 have priority:1 and hidden:false, while dbslave01 has priority:0 and hidden:true.
After migrating the whole cluster to MongoDB3.0 the replication statuses are: mongo02 - PRIMARY, mongo01 - SECONDARY. while dbslave01 is still in "not reacheable/healthy" state, because it crashed during initial startup (though that's not the point of this ticket).

So having that system state the problem is:
I log in into mongo01 (SECONDARY) and issue a db.shutdownServer() command. That works fine on mongo01 and it goes down. But at the same time mongo02 (PRIMARY) triggers a reelection, and election fails because it thinks it needs to have at least two votes to become primary again, while dbslave01 is unable to vote (being down) and thus one vote is missing.

That seems like a errorneous behaviour since unreacheable system should obviously be not expected to vote and the only live server should happily reelect itself to keep being primary. Or ideally election would not trigger because of a secondary shutdown if a primary is still live and serving, because even successful election will still cause connections dropping and service interuption.



 Comments   
Comment by Scott Hernandez (Inactive) [ 09/Jun/15 ]

Alex, what you describe is a set of 3 voting nodes where 2 are down, and cannot vote. In this scenario 1/3 of the votes is not a majority and cannot elect a primary, by design. If that one node, alone, could become primary then it would lead to problems where split networks or other failure cases could cause more than 1 primary to exist, for example.

Here are the docs discussing this in a bit more detail which you might find helpful: http://docs.mongodb.org/manual/core/replica-set-elections/

Generated at Thu Feb 08 03:49:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.