We have recently moved to a MongoDB cluster setup from a single node setup.

We have the dynamic network setup where the DNS entries activate after about 30 seconds a service(MongoDB Node) is started. Each service/node has a unique IP allocated after start/restart.

Lets say my Mongo set up has 3 Nodes rs1,rs2,rs3.. Here is whats happening..

1. When i shutdown rs1,rs2 and rs3 together, the DNS records for are removed.
2. After restart, each MongoDB node does a isSelf test, to find its hostname. isSelf test is simply MongoDB node connecting to each of the members in replica set config to find itself.
3. Due to above network setup delays isSelf test fails and the node enters REMOVED state.

{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"NETWORK", "id":4834700, "ctx":"ReplCoord-0","msg":"isSelf could not connect via connectSocketOnly","attr":\{"hostAndPort":"rs1.example.com:12000","error":{"code":6,"codeName":"HostUnreachable","errmsg":"couldn't connect to server rs1.example.com:12000, connection attempt failed: HostNotFound: Could not find address for rs1.example.com:12000: SocketException: Host not found (authoritative)"}}}
{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"REPL", "id":21394, "ctx":"ReplCoord-0","msg":"This node is not a member of the config"}
{"t":{"$date":"2022-01-18T15:57:16.513+05:30"},"s":"I", "c":"REPL", "id":21358, "ctx":"ReplCoord-0","msg":"Replica set state transition","attr":{"newState":"REMOVED","oldState":"STARTUP"}}

4. Once a node enters REMOVED state it does not try to find other nodes and initiate cluster formation/election. So, MongoDB cluster formation fails after each restart of all Nodes.

I think dynamic DNS environments like ours are seen in other Organizations also. Isn't it better to keep retrying isSelf test even after entering REMOVED state?