[CSHARP-1294] Fail to connect if seed name is different than host name Created: 22/May/15  Updated: 22/May/15  Resolved: 22/May/15

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: 2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: tryshi Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Repro:

In a replication set, mongodb service is listening with hostname(which resolves to a private IP in a virtual network).
From client side, the seed list is composed of public DNS names.
then C# driver fails to connect, with below error:

Error:

     at MongoDB.Driver.Core.Servers.ClusterableServer.<HeartbeatAsync>d__d.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : {color:red}"Unspecified/vmsec0521e:27000"{color} }", EndPoint: "Unspecified/vmsec0521e:27000", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: No such host is known
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)

Code:

            var cnnString = "mongodb://<userName>:<pwd>@vmpri0521e.westus.cloudapp.azure.com:27000, {color:red}vmsec0521e.westus.cloudapp.azure.com:27000{color}/testoc?replicaSet=rs0521e";
            var client = new MongoClient(cnnString);
            var db = client.GetDatabase("testoc");
            var col = db.GetCollection<Post>("Posts");
            var posts = await col.Find(e => true).ToListAsync();

rs.conf information

rs0521e:PRIMARY> rs.conf()
{
        "_id" : "rs0521e",
        "version" : 7,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "vmpri0521e:27000",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 5,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "vmsec0521e:27000",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "vmarb0521e:27000",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 0,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatTimeoutSecs" : 10,
                "getLastErrorModes" : {
 
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                }
        }
}

rs.status information:

rs0521e:PRIMARY> rs.status()
{
        "set" : "rs0521e",
        "date" : ISODate("2015-05-22T18:18:20.610Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "vmpri0521e:27000",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 10942,
                        "optime" : Timestamp(1432308408, 1),
                        "optimeDate" : ISODate("2015-05-22T15:26:48Z"),
                        "electionTime" : Timestamp(1432307801, 1),
                        "electionDate" : ISODate("2015-05-22T15:16:41Z"),
                        "configVersion" : 7,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "vmsec0521e:27000",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 10292,
                        "optime" : Timestamp(1432308408, 1),
                        "optimeDate" : ISODate("2015-05-22T15:26:48Z"),
                        "lastHeartbeat" : ISODate("2015-05-22T18:18:20.351Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-22T18:18:19.195Z"),
                        "pingMs" : 1,
                        "configVersion" : 7
                },
                {
                        "_id" : 2,
                        "name" : "vmarb0521e:27000",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 10292,
                        "lastHeartbeat" : ISODate("2015-05-22T18:18:20.026Z"),
                        "lastHeartbeatRecv" : ISODate("2015-05-22T18:18:18.991Z"),
                        "pingMs" : 1,
                        "configVersion" : 7
                }
        ],
        "ok" : 1
}

More Information:

Not repro in NodeJs driver with same connection string. So looks like this is a C# driver issue.



 Comments   
Comment by Craig Wilson [ 22/May/15 ]

Yes, we can certainly keep polling the public DNS name. Let's take that and run with it.

1. Our seedlist contains publicHostA, publicHostB, and publicHostC.
2. Our repl set config contains privateHostA, privateHostB, and privateHostC;
3. We remove privateHostA from the repl set.
4. Drivers notice that privateHostA is gone, but can't do anything about it because we don't know that publicHostA and privateHostA are really the same. We also notice that publicHostA isn't responding, but these two events may or may not be linked... we can't know. Depending on order, we might notice that publicHostA is no longer part of the repl set, and can drop it at that point.
5. We add privateHostD to the repl set.
6. Drivers notice that privateHostD has been added, but we can't talk to it... ever.

The spec is just about conformity between the drivers and assures us that we are doing the right thing. In this case, there isn't another option. As illustrated in (4) and (6), the driver is effectively helpless. I understand the expectations thing and totally agree, but this particular case doesn't get to that ask that question. We do what we can do.

However, we can try and figure out a way for you as a user to supply a mapping to us via code. We could do it via callback, so you might be able to query a configuration service or simply know about a well known translation between private host and public host names. But that's the best we'll be able to do here. If you think that it is worthwhile to pursue the "via code etc/hosts" configuration option, let's open another ticket to track that feature request. For now, I'm going to close this ticket as Works as Designed.

Comment by tryshi [ 22/May/15 ]

Thanks Craig for the detailed explanation.
I'm not sure about the spec, but in my opinion, if something is not expected by the customers, we should consider fixing the spec.

I don't know much about mongodb internals, but just a naive answer to your below question:
>> if you removed vmsec0521e:27000 from your replica set, how would we know that?
Can we keep polling the public DNS name (or what ever is in the seed list) to get current cluster configuration? so we know the changes to a replicate set.

Also thanks for the workaround suggestion, that unblocks me now, but it will be a costly maintenance work.

Comment by Craig Wilson [ 22/May/15 ]

Hi tryshi,

I can't speak to the node driver, but there is a specification that exists which indicates our current behavior is correct. We take your seed list and use it to connect to the nodes initially. We then hear back from them and notice that none of the nodes in the seed list is what is configured in the replica set config, remove your seed list nodes and add the replica set nodes. If we don't do this, we couldn't respond to replica set changes. For instance, if you removed vmsec0521e:27000 from your replica set, how would we know that?

Therefore, it is required that your client be able to resolve the hostnames as defined in the replica set configuration. There are certainly some ways to handle this. The easiest is to add entries to your etc/hosts file that will resolve those private dns names to their public dns equivalent. The problem here is that when you add a new node, you'll need to add that new node's entry into all your hosts files.

I'm reading the documentation on our mongodb docs site and this is stated:

Finally ensure that each member of a replica set is accessible by way of resolvable DNS or hostnames. You should either configure your DNS names appropriately or set up your systems’ /etc/hosts file to reflect this configuration.

Does that explain it? Do you have any further questions?
Craig

Comment by tryshi [ 22/May/15 ]

This repros in the C# Driver 2.0

Comment by tryshi [ 22/May/15 ]

Note:

The client is not in the virtual network, so has no access to the private ip in the virtual network.

Generated at Wed Feb 07 21:39:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.