[JAVA-560] Mongo and Mongo.Holder internal pooling don't tolerate IP address changes Created: 24/Apr/12  Updated: 25/Jun/13  Resolved: 25/Jun/13

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: 2.7.2
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Jared D. Cottrell Assignee: Unassigned
Resolution: Duplicate Votes: 9
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Java 1.6 on Ubuntu and Mac OS X, MongoDB 2.0.4 for Ubuntu and Mac OS X, 64-bit, running in auth mode


Issue Links:
Duplicate
duplicates JAVA-707 Refresh the IP resolved for hosts fre... Closed

 Description   

For singleton Mongos and Mongos managed by Mongo.Holders a change in a server IP address causes errors when trying to use the objects to make further queries. There seems to be some code in place to refresh pooled replica set connections, though I haven't tested that it works.

Note that in both cases we are using MongoURIs to connect.

To replicate we added the IpTest class to our app server (code below), fronted by a simple JSP that calls findStuff() with the result of getStuffCollection on each of the static member variables from the Mongo URI string down to the DBCollection itself. We then:

1. Started the app
2. Hit the JSP page (all methods passed without error)
3. Changed the IP address of the machine (in this case by switching to another wireless network)
4. Hit the JSP page

All but the Mongo URI and Mongo URI string failed with the same operation timeout exception. Since IP addresses change fairly frequently in the cloud we have now started running into problems in production where we frequently need to restart the application process as a result of this.

package com.company.iptest;
 
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.Mongo;
import com.mongodb.MongoURI;
import com.mongodb.ServerAddress;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import java.net.UnknownHostException;
 
public class IpTest {
 
    private static final Logger log = LoggerFactory.getLogger(IpTest.class);
 
    public static final String _mongoUriString = "mongodb://username:password@localhost:27017/database";
 
    public static MongoURI _mongoUri;
    public static Mongo.Holder _mongoHolder;
    public static Mongo _mongo;
    public static DB _db;
    public static DBCollection _collection;
 
    static {
        try {
            _mongoUri = new MongoURI(_mongoUriString);
            _mongoHolder = new Mongo.Holder();
            _mongo = _mongoHolder.connect(_mongoUri);
            _mongo.getDB(_mongoUri.getDatabase()).authenticate(_mongoUri.getUsername(), _mongoUri.getPassword());
            _db = _mongo.getDB("database");
            _collection = _db.getCollection("stuff");
        } catch ( Exception e ) {
            log.error("Unexpected exception", e);
        }
    }
 
    public static DBCollection getStuffCollection ( String mongoUriString )
            throws UnknownHostException {
        return getStuffCollection(new MongoURI(mongoUriString));
    }
 
    public static DBCollection getStuffCollection ( MongoURI mongoUri )
            throws UnknownHostException {
        DBCollection c = getStuffCollection(new Mongo(mongoUri));
        DB db = c.getDB();
        if ( !db.isAuthenticated() ) {
            db = db.getMongo().getDB(mongoUri.getDatabase());
            db.authenticate(mongoUri.getUsername(), mongoUri.getPassword());
        }
        return c;
    }
 
    public static DBCollection getStuffCollection ( Mongo.Holder mongoHolder )
            throws UnknownHostException {
        return getStuffCollection(mongoHolder.connect(new MongoURI(_mongoUriString)));
    }
 
    public static DBCollection getStuffCollection ( Mongo mongo ) {
        return getStuffCollection(mongo.getDB("database"));
    }
 
    public static DBCollection getStuffCollection ( DB db ) {
        return db.getCollection("stuff");
    }
 
    public static DBObject findStuff ( DBCollection stuffCollection ) {
        return stuffCollection.findOne();
    }
 
}

Caused by: com.mongodb.MongoException$Network: can't call something : mymac.local/10.0.0.5:27017/database
	at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:227)
	at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:305)
	at com.mongodb.DBCollection.findOne(DBCollection.java:332)
	at com.mongodb.DBCollection.findOne(DBCollection.java:318)
        at [ ... application code ... ]
	... 64 more
Caused by: java.net.SocketException: Operation timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	at org.bson.io.Bits.readFully(Bits.java:35)
	at org.bson.io.Bits.readFully(Bits.java:28)
	at com.mongodb.Response.<init>(Response.java:39)
	at com.mongodb.DBPort.go(DBPort.java:128)
	at com.mongodb.DBPort.call(DBPort.java:79)
	at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:218)
	... 69 more



 Comments   
Comment by Jeffrey Yemin [ 25/Jun/13 ]

Duplicate JAVA-707

Comment by Jared D. Cottrell [ 24/Apr/12 ]

Yeah, saw the code that spawns a thread to re-resolve replica set connections. Unfortunately, these connections aren't replica set connections. This ticket is about the singleton connections. (Not sure what the proper name is for non-rs connections.)

Comment by Scott Hernandez (Inactive) [ 24/Apr/12 ]

I think you want this setting

You can decrease the "com.mongodb.inetAddrCacheMS" system property to affect how often addresses are resolved via replica set discovery/health checks.

If you aren't using a replica set connection then connections are created in the pool and resolution is done when the socket/connection is established I believe – would need to double check to be sure.

Comment by Jared D. Cottrell [ 24/Apr/12 ]

I think I see what you're saying now. Let's call that a red herring, at least as far as this issue is concerned. The fact that it's not using the loopback address could be filed as a separate issue for a performance boost. In production where the real problem is happening the URIs aren't using localhost but referring to remove servers.

Comment by Jared D. Cottrell [ 24/Apr/12 ]

Not sure I follow. localhost resolves to the IP I'm forcing it to and intentionally changing mid-test. I'm doing this by switching networks. And the networks are entirely different subnets. One is on 10.0.0.0/255.0.0.0 and the other is on 192.168.1.0/255.255.255.0.

However the URI is using "localhost" and not the IPs. So I would expect that especially those methods that get a fresh crack at the URI each time would notice the IP address change.

Comment by Scott Hernandez (Inactive) [ 24/Apr/12 ]

What does localhost resolve to on that machine? It looks like it not using the correct loopback address (127.0.0.1).

Generated at Thu Feb 08 08:52:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.