[SERVER-35649] Nodes removed due to isSelf failure should re-attempt to find themselves Created: 18/Jun/18 Updated: 08/Jan/24 Resolved: 21/Sep/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Networking, Replication |
| Affects Version/s: | 3.4.13 |
| Fix Version/s: | 4.8.0, 4.4.2, 4.2.13, 4.0.24 |
| Type: | New Feature | Priority: | Minor - P4 |
| Reporter: | Owen Allen | Assignee: | A. Jesse Jiryu Davis |
| Resolution: | Fixed | Votes: | 3 |
| Labels: | former-quick-wins, gm-ack | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v4.4, v4.2, v4.0
|
||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Repl 2020-08-24, Repl 2020-09-07, Repl 2020-09-21, Repl 2020-10-05 | ||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 40 | ||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
We have some dev/staging environments which are locally hosted in our office building. They are entirely for internal usage, so there uptime isn't critical. We have recently been experiencing power outages that cause all 3 members of the replica set to go down and then when power restores come back up at the same time. This has happened about 5 times now and each time when the replica set comes back up both the primary/secondary end up in the REMOVED status and never recover unless we manually restart one of the mongo processes. mongo-dev1 rs.status()
mongo-dev2 rs.status()
mongo-dev1 show log rs
mongo-dev1 rs.conf()
As I read the documentation I can't find much information about the REMOVED status. In our setup, since mongo-dev1 has a priority of 2 and mongo-dev2 has a priority of 1 I would expect mongo-dev1 to be elected as the primary after the reboot. Is this a bug or are we doing something wrong. If it's a bug, what is the proper procedure for this when all members of the replica set come online at the same time? |
| Comments |
| Comment by Githook User [ 11/Mar/21 ] |
|
Author: {'name': 'Xuerui Fa', 'email': 'xuerui.fa@mongodb.com', 'username': 'XueruiFa'}Message: (cherry picked from commit 418c61279986a5eaddc66a16b5e288556ad1f6d3)
(cherry picked from commit 28c6948a2a02760a69aaee3875c4b2a427528a5a) |
| Comment by Steven Vannelli [ 01/Mar/21 ] |
|
Requesting a 4.0 backport to resolve BF-20251 |
| Comment by Githook User [ 04/Feb/21 ] |
|
Author: {'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}Message: (cherry picked from commit 418c61279986a5eaddc66a16b5e288556ad1f6d3) |
| Comment by Githook User [ 15/Oct/20 ] |
|
Author: {'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}Message: (cherry picked from commit 418c61279986a5eaddc66a16b5e288556ad1f6d3)
(cherry picked from commit 28c6948a2a02760a69aaee3875c4b2a427528a5a) |
| Comment by Githook User [ 21/Sep/20 ] |
|
Author: {'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}Message: |
| Comment by Nick Brewer [ 29/Jun/18 ] |
|
Since DNS infrastructure typically has some level of geographic redundancy, it's not a hard requirement that replica set nodes be able to resolve themselves without DNS - however it's necessary in a case like this where the DNS resolvers are coming online at roughly the same time as (or perhaps slightly after) the mongod nodes.
I'm passing this along to our replication team as a feature request. Regards, |
| Comment by Owen Allen [ 25/Jun/18 ] |
|
In that situation, it seems like it would be considered best practice (possibly called out in the docs) that replset members should be able to locate themselves without DNS. Is it possible to have nodes that are in REMOVED state, especially if it's due to isSelf failure, check back rather than requiring manual intervention? |
| Comment by Spencer Brody (Inactive) [ 25/Jun/18 ] |
|
The first time a mongod logs a new replica set config, whether from a reconfig or at startup when loading the config from the disk, the first thing it does is try to locate itself in the config. In order to do this, it iterates through every member in the config and runs a simple command (called 'isSelf') against each member, with a unique ID to identify that it is the sender. When the node receives an isSelf command that it can determine was from itself because it contains the same unique ID that it sent, it then knows which name in the config corresponds to itself. If the node can't resolve its own hostname during this process, then the node will never be able to send the command to itself, and will thus conclude that it isn't part of the config, and will then transition to the REMOVED state. At that point it won't re-attempt to find itself in the config until it is restarted, or it receives a new config via a replSetReconfig. |
| Comment by Nick Brewer [ 25/Jun/18 ] |
This is a good point - in most of the instances where I've seen a node enter the REMOVED state, it's simply due to its inability to connect to other members of a set, often due to a change in IP address or port. A DNS blackout would have a similar effect. To test this a bit more, you could take your resolvers out of the equation entirely and hard-code the correct entries into /etc/hosts, assuming that your nodes have static addresses. I suspect that when the addresses are added in this way, the replica set members will be able to find each other on simultaneous startup. Nick |
| Comment by Owen Allen [ 25/Jun/18 ] |
|
Unfortunately I was too slow to respond and our log files have rolled since then. One of our dev's theory was that it had to do with the mongo boxes coming up at the same time that our DNS server came up (since the power outage claimed it too), and since the initial DNS queries would have failed since our replset uses hostname rather than IPs, it affects the repl set config in a way that it doesn't recover from. From my local tests this appears to be replicate the existing condition. I have attached a log file from that test. The first few reboots were down trying to replicate all 3 boxes coming up at once (without DNS interruption). In those cases they booted up with no issue. In the last case, I altered the local resolv.conf so that it was no longer able to locate intranet DNS entries (such as the other members of the replset). If you look for when the REMOVED statement is you'll see the before and after chatter. Now, if I correct the resolv.conf and am able to locate intranet resources, Mongo does not correct the issue, it remains in REMOVED status forever. Basically, this signals to me that if Mongo boots up when DNS is down, it can remain broken forever. Now, reading the log entry, perhaps I am setting up the mongo box incorrectly since it states, "2018-06-25T18:10:38.907+0000 I REPL [replExecDBWorker-1] This node is not a member of the config". Perhaps I need to ensure that the boxes are able to locate themselves even if DNS is down. mongod.log |
| Comment by Nick Brewer [ 19/Jun/18 ] |
|
Hi, I'd like to see the full log messages starting from the time that the machines are brought back up. Thanks, |