[JAVA-310] Driver reconnect does not detect network IP changes. Created: 28/Mar/11  Updated: 03/Dec/14  Resolved: 12/Jul/13

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: None
Fix Version/s: 2.12.0, 3.0.0

Type: Bug Priority: Major - P3
Reporter: Leif Mortenson Assignee: Jeffrey Yemin
Resolution: Done Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Mac OS X
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-9M3263)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)


Issue Links:
Duplicate
is duplicated by JAVA-707 Refresh the IP resolved for hosts fre... Closed
is duplicated by JAVA-1582 MongoClient constructor throws except... Closed
Related
related to CSHARP-869 Stop Caching the IPEndPoint for a Ser... Closed
is related to JAVA-249 Java Driver cannot connect to 'localh... Closed
is related to JAVA-1582 MongoClient constructor throws except... Closed

 Description   

Our application runs on machines that are sometimes moved between networks or has its IP address changed.

When we create the Mongo instance it is done using the "localhost" host for the Mongo server. The Mongo driver appears to always resolve to an IP on the local subnet. "192.168.0.10" for example. When the network chances however, our applications starts failing with errors like the following because it can no longer connect:

jvm 1 | java.io.IOException: couldn't connect to [io/192.168.0.10:27017] bc:java.net.ConnectException: Operation timed out
jvm 1 | at com.mongodb.DBPort._open(DBPort.java:205)
jvm 1 | at com.mongodb.DBPort.go(DBPort.java:94)
jvm 1 | at com.mongodb.DBPort.go(DBPort.java:75)
jvm 1 | at com.mongodb.DBPort.call(DBPort.java:65)
jvm 1 | at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:215)
jvm 1 | at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)
jvm 1 | at com.mongodb.DBCursor._check(DBCursor.java:354)
jvm 1 | at com.mongodb.DBCursor._hasNext(DBCursor.java:484)
jvm 1 | at com.mongodb.DBCursor.hasNext(DBCursor.java:509)
jvm 1 | at com.mongodb.jdbc.MongoResultSet.next(MongoResultSet.java:640)
...

I have configured the Mongo instance to automatically try to reconnect, but the conversion from "localhost" to "192.168.0.10" only happens once. The only way to get Mongo to reconnect is to restart the Java instance.

I assume that there is a reason why the IP is being resolved, but would it be possible to make the driver re-resolve the IP when doing a reconnect?

Thanks in advance,
Leif



 Comments   
Comment by Jeffrey Yemin [ 27/Feb/14 ]

Hi there,

If you would like to test the fix for this issue, 2.12.0-rc0 is available either on github or Maven Central.

Thanks,
Jeff

Comment by auto [ 09/Jul/13 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Message: JAVA-310: Removed caching of InetSocketAddress in ServerAddress
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/0c0d94a8efd9fad3248316b8dfd03676b7f97174

Comment by Jeffrey Yemin [ 25/Jun/13 ]

Fixed by design in 3.0.x branch. ServerAddress is immutable and no longer caches InetSocketAddress. See

https://github.com/mongodb/mongo-java-driver/blob/3.0.x/driver/src/main/org/mongodb/connection/ServerAddress.java#L155

Comment by Heiko Hahn [ 19/Jun/13 ]

This still happens with java mongo driver 2.9.3 on Linux (AWS)
We are initializing the connection with symbolic names to mongos instances.
After exchanging / moving a mongos instance to a new EC2 instance (same cname, different internal IP), we get "Server seen down" messages with the correct name, but the old IP in the logs.

Only way to fix this was to restart the java process.

Best,
Heiko

Comment by Jeffrey Yemin [ 15/May/12 ]

The localhost issue is resolved in JAVA-249. There is still an issue if the IP address of a non-local server is changed.

Comment by Leif Mortenson [ 29/Mar/11 ]

Antoine,
Thank for your attention. I just posted here because it was unintuitive how it was working. If you think there is a way to make it work a little better when using "localhost" that would be great. For me, I am happy with using the localhost IP address directly. So feel free to close this off.

Thanks for your help
Leif

Comment by Antoine Girbal [ 29/Mar/11 ]

if you use only 1 server host (either localhost or 127.0.0.1) then the driver will be in mode "direct", not "replica set".
Hence it will only use the local server, instead of detecting and using the master.
To use replica set you must provide a list of servers (ideally at least 2 servers) and use the hosts as they are defined in the replica set configuration.
For any further issue on setup or if you just need advice, you should use the google groups.
As for localhost resolving on mac, we may try to improve it to use 127.

Comment by Leif Mortenson [ 29/Mar/11 ]

Antoine,
Understood. Thank you for looking into it. I am able to get things working on my machines using the "127.0.0.1" work around.

I had found that code and did try running InetAddress.getLocalHost().getHostAddress() on my system but that also returns "127.0.01" in the same VM. So I didn't think that was what was happening. I didn't dig into the code further.

Unrelated, but is there any particular reason why localhost shouldn't be used? We are setting up a clustered application of 3 servers where each server has a mongo instance as well as a Java based web app. They each point to their localhost mongo and then let the driver and mongo work out which is the master. We can then do load balancing between the web apps. This helps keep the system simple and also helps reduce hw costs initially. Initially there is no sharding, just replication between the 3 mongos for failover and read load balancing of the system.

Thanks,
Leif

Comment by Antoine Girbal [ 28/Mar/11 ]

I think the problem is that we added a workaround for resolving localhost on Mac OS, because it could not resolve properly.
The code to do that is:

if ( host.toLowerCase().equals("localhost") ){
return new InetAddress[]

{ InetAddress.getLocalHost()}

;
}

The way it works is that it resolves the hostname of the server, not "localhost", and hence gets the external IP address.
It is fairly rare that users have the driver on same machine as mongod, besides when doing testing.
Also, driver is able to update IP address of hosts, but only in replica set mode, and it takes a few minutes to change.
Agreed it can lead to unexpected behavior but we dont want to do OS detection for Mac.

Comment by Leif Mortenson [ 28/Mar/11 ]

Scott,
Thank you for the quick reply. I am not doing anything intentionally on this machine to map to anything other than 127.0.0.1.

Just to check:
1) From the shell:

leif$ host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1

2) From within the same JVM, when I run the following code:

InetAddress lh = InetAddress.getByName( "localhost" );
System.out.println( "ip: " + lh.getHostAddress() );

I get this:

ip: 127.0.0.1

3) The hosts file has the following:

127.0.0.1 localhost
::1 localhost
fe80::1%lo0 localhost

This all looks correct which is why I was thinking that Mongo is doing something to go out of its way to resolve the IP address.

If I use the host name "127.0.0.1" rather than "localhost" then everything works as I need. So this is a workaround for now. But it is strange that the two would work differently.

Thanks,
Leif

Comment by Scott Hernandez (Inactive) [ 28/Mar/11 ]

Why is your machine setup to resolve localhost to something other than 127.0.0.1? Why not just use 127.0.0.1 as the address you connect to?

There are performance reasons why it is best not to resolve the address on every use; also with a connection pool it can be complicated but it would make sense to call ServerAddress.updateInetAddr() (which reresolves the name) if there are no sockets in the pool.

Generated at Thu Feb 08 08:51:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.