[SERVER-10364] Split Brain when the Primary loses the majority of the cluster, but the cluster can still see it. Created: 28/Jul/13 Updated: 21/Aug/15 Resolved: 29/Jul/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.4.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Michael Tewner | Assignee: | Matt Dannenberg |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
ProofOfConcept: Node#1 - Primary, Node#2 - Secondary and Arbiter |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | I was able to reproduce the situation where: I simulated this by putting the Secondary and the arbiter on one server, and the Primary on a different server. |
||||||||
| Participants: | |||||||||
| Description |
|
Last week, there was a failure of AWS DNS resolution which caused a specific Amazon Availability Zone to not be able to resolve DNS. Other AZ's WERE able to resolve DNS, including records of hosts in the "DNS-failed" zone. In a nutshell, we have the following situation which led to both nodes in "SECONDARY" state: PRIMARY (db01srv02) - suddenly can't see the SECONDARY or the ARBITER. It steps down. (n.b. - after upgrading to 2.4.5, I now get the more descriptive error "Sun Jul 28 12:43:36 [rsMgr] not electing self, db01srv02.local.:20001 would veto with 'I don't think db01srv01.local.:10001 is electable'" Disclaimer - I'm not a DB Expert, so this may be expected behavior for some reason.... |
| Comments |
| Comment by Michael Tewner [ 31/Jul/13 ] |
|
Thank you very much, Matt! |
| Comment by Matt Dannenberg [ 30/Jul/13 ] |
|
All nodes visible to the candidate take part in the election. The algorithm does not account for a pair of nodes that have only one direction of visibility. I've created a ticket for this bug, which you can view and track the progress of: |
| Comment by Michael Tewner [ 30/Jul/13 ] |
|
Again, I'm not a DB expert, but why does a fenced node (the old primary can't see the majority of the cluster) get a say in the elections? What do I care if the old primary can't see the primary-to-be when it can't see half of the cluster anyway? It seems as though this one-way DNS resolution failure has caused an asymmetric issue - In a normal network segmentation failure event, the old primary would ALWAYS veto the new primary (IIUnderstandC), but usually the the primary-elect is unable to connect to ("ask") it in the first place. In our failure, the primary-elect WAS able to query the secondary which gave what seems to be an "uninformed" response (as it was a fenced node). |
| Comment by Matt Dannenberg [ 29/Jul/13 ] |
|
You are correct; what you are seeing is correct, intended behavior. The server that was PRIMARY should step down when it can't see the majority of the servers as they (in normal operation) cannot see the PRIMARY and would elect a new one. The server that started as a SECONDARY asks the former PRIMARY if the SECONDARY's address can become PRIMARY and the former PRIMARY says it would veto because it cannot see the server at that address and believes it to be DOWN as a result, so the SECONDARY doesn't start an election. |
| Comment by Michael Tewner [ 28/Jul/13 ] |
|
Also, I probably should not have called this a "split brain", as both of the nodes are left SECONDARY state, not PRIMARY. |
| Comment by Michael Tewner [ 28/Jul/13 ] |
|
I should expand on my test-plan - After building a cluster between the 2 server (SECONDARY and ARBITER on one server, PRIMARY on the other), I remove the PRIMARY:/etc/hosts entries for the SECONDARY and ARBITER hosts, simulating the loss of DNS resolution for that node. Again, the SECONDARY and ARBITER still have /etc/hosts records for the PRIMARY (simulating that DNS still works for them). |
| Comment by Michael Tewner [ 28/Jul/13 ] |
|
Same test using 2.4.5. |