[SERVER-62699] Replica set fails to restart after shutdown of all Nodes in a Dynamic DNS/network environment Created: 18/Jan/22  Updated: 30/May/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: 4.4.14, 4.4.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sravan _ Assignee: Alan Zheng
Resolution: Unresolved Votes: 1
Labels: pmr
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive data.zip     Zip Archive logs.zip     Text File mongo_replicaset_error_reproduce_steps.txt    
Issue Links:
Related
related to SERVER-35649 Nodes removed due to isSelf failure s... Closed
is related to SERVER-35649 Nodes removed due to isSelf failure s... Closed
is related to SERVER-40159 Add retry logic for name resolution f... Closed
Operating System: ALL
Steps To Reproduce:
  1. Stop all nodes in a MongoDB replica setup.
  2. Remove DNS entries related to the MongoDB nodes.
  3. Start all MongoDB nodes in replica setup.
  4. Nodes will fail isSelf test and enter REMOVED state.
  5. Add the relevant DNS entries in your network.
  6. MongoDB nodes still remain in REMOVED state and does not retry isSelf test.
Participants:

 Description   

We have recently moved to a MongoDB cluster setup from a single node setup.

We have the dynamic network setup where the DNS entries activate after about 30 seconds a service(MongoDB Node) is started. Each service/node has a unique IP allocated after start/restart.

Lets say my Mongo set up has 3 Nodes rs1,rs2,rs3.. Here is whats happening..

1. When i shutdown rs1,rs2 and rs3 together, the DNS records for are removed.
2. After restart, each MongoDB node does a isSelf test, to find its hostname. isSelf test is simply MongoDB node connecting to each of the members in replica set config to find itself.
3. Due to above network setup delays isSelf test fails and the node enters REMOVED state.

{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"NETWORK", "id":4834700, "ctx":"ReplCoord-0","msg":"isSelf could not connect via connectSocketOnly","attr":\{"hostAndPort":"rs1.example.com:12000","error":{"code":6,"codeName":"HostUnreachable","errmsg":"couldn't connect to server rs1.example.com:12000, connection attempt failed: HostNotFound: Could not find address for rs1.example.com:12000: SocketException: Host not found (authoritative)"}}}
{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"REPL", "id":21394, "ctx":"ReplCoord-0","msg":"This node is not a member of the config"}
{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"REPL", "id":21358, "ctx":"ReplCoord-0","msg":"Replica set state transition","attr":{"newState":"REMOVED","oldState":"STARTUP"}}

4. Once a node enters REMOVED state it does not try to find other nodes and initiate cluster formation/election. So, MongoDB cluster formation fails after each restart of all Nodes.

I think dynamic DNS environments like ours are seen in other Organizations also. Isn't it better to keep retrying isSelf test even after entering REMOVED state?

 



 Comments   
Comment by Sravan _ [ 09/Jan/23 ]

Hi @Alan Zheng, Is there any timeline when this issue is expected to be fixed?

 

Comment by Sravan _ [ 22/Jun/22 ]

However, if at least ONE node's DNS works, it will restore the replica set config as soon as the DNS returns on the remaining nodes.

Got it Christopher..
But In my case, I  cannot control the dynamic DNS setup delay when mongo nodes restart..

Comment by Chris Kelly [ 13/Jun/22 ]

Hi _Sravan,

Thanks for your report, and taking the time to submit your logs and reproduction. I was able to reproduce this behavior on 4.4.14. I created a 3 node replica set and created 3 hosts in /etc/hosts for them to use. I shut down the replica set, temporarily removed the entries from /etc/hosts, restarted the replica set, and observed the (expected) InvalidReplicaSetConfig in rs.status().

I then re-introduced the entries to /etc/hosts while the replica set was still running and waited to see if it would resolve - it did not.

However, if at least ONE node's DNS works, it will restore the replica set config as soon as the DNS returns on the remaining nodes. So, if you were to leave 1 entry in /etc/hosts when starting up the whole replica set, that node will start up successfully while the others will still have an rs.status() of InvalidReplicaSetConfig. As soon as the other entries are returned to /etc/hosts, the config will immediately resolve on the other nodes and they will all be present in rs.status().

It seems like this issue was looked at before in SERVER-35649, but may not affect the case where every single node's DNS fails during startup. I'll pass this along to the replication team to see if they may have input on this. I noticed a potentially similar issue in SERVER-40159 as well.

Christopher

Comment by Ricardo Ferreira [ 19/May/22 ]

Hi,

I'm deploy in Azure and I got this error when a add a readinessProbe. 
My readiness Probe is like this.

        readinessProbe:
          exec:
            command:
              - mongo
              - --disableImplicitSessions
              - --eval
              - "db.adminCommand('ping')"
          initialDelaySeconds: 120
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 6

 
If I remove readinessProbe it always work well.

Comment by Sravan _ [ 18/May/22 ]

Hi @Brooke and @Eric,

I created a PoC database on my dev server to just reproduce this issue..

Attached the entire data directory archive, logs archive and commands/steps used to reproduce the issue..

data.zip logs.zip mongo_replicaset_error_reproduce_steps.txt

Thanks,
Sravan

Comment by Brooke Miller [ 12/May/22 ]

Hi g.sravan4u@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please reply with the additional information (requested in Eric's previous comment)?

Thanks,

Brooke 

Comment by Eric Sedor [ 07/Mar/22 ]

Thanks g.sravan4u@gmail.com, I did mean SERVER-35649. It sounds possible that the fix for SERVER-35649 isn't helpful here because in the scenario you're testing DNS for all nodes are being removed. My theory is that the heartbeats that would trigger another lookup can't happen.

Are you able to perform this test and then provide, for all nodes, an archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here), and mongod logs, and attach them to this ticket?

Comment by Sravan _ [ 20/Jan/22 ]

Hi Eric, I think you mean this seems similar to https://jira.mongodb.org/browse/SERVER-35649

We are using stable release version 4.4.7

Comment by Eric Sedor [ 19/Jan/22 ]

Thank you for your detailed walkthrough g.sravan4u@gmail.com,

Can I please ask that you submit this same request to [feedback.mongodb.com|feedback.mongodb.com]? We're starting to direct new feature and improvement requests to that channel and preferring this JIRA project for bug reports specifically.

You may also want to search and post on the MongoDB Developer Community Forums, as it's possible there are others who have guidance on how to satisfy your use-case.

EDIT: I misread this originally, sorry, and want to shift to asking what version of MongoDB this is on; as it seems similar to SERVER-62699

Thank you,
Eric

Generated at Thu Feb 08 05:55:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.