[JAVA-1432] Auto Reconnect Created: 08/Sep/14  Updated: 11/Sep/19  Resolved: 23/Jun/15

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: 2.12.3
Fix Version/s: None

Type: Task Priority: Trivial - P5
Reporter: Mohamed Nouh Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File MongoDataStore.java     Java Source File MongoTest.java    

 Description   

I'm currently using the java driver, we've noticed that this is not auto reconnecting to mongodb by default. The below says autoConnectRetry has been deprecated, how can I use the connectTimeout property to have auto retry?

@Deprecated
public MongoClientOptions.Builder autoConnectRetry(boolean autoConnectRetry)

Deprecated. There is no replacement for this method. Use the connectTimeout property to control connection timeout.
Sets whether auto connect retry is enabled.

Parameters:
autoConnectRetry - auto connect retry



 Comments   
Comment by Jeffrey Yemin [ 23/Sep/14 ]

I think this last exception is a Mule issue, not a Java driver issue. See for example https://www.mulesoft.org/jira/browse/MULE-6085.

Comment by Mohamed Nouh [ 23/Sep/14 ]

About 10 seconds before the restart of the server. I will run some more tests. In addition, if we start the application while Mongo is unavailable, we currently receive this error repeatedly:

Sep 22, 2014 8:34:38 PM com.mongodb.ServerMonitor$ServerMonitorRunnable run
WARNING: Exception in monitor thread during notification of server state change
java.lang.IllegalStateException: zip file closed
	at java.util.zip.ZipFile.ensureOpen(ZipFile.java:634)
	at java.util.zip.ZipFile.getEntry(ZipFile.java:305)
	at java.util.jar.JarFile.getEntry(JarFile.java:226)
	at java.util.jar.JarFile.getJarEntry(JarFile.java:209)
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:840)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at org.mule.module.launcher.FineGrainedControlClassLoader.findClass(FineGrainedControlClassLoader.java:175)
	at org.mule.module.launcher.MuleApplicationClassLoader.findClass(MuleApplicationClassLoader.java:134)
	at org.mule.module.launcher.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:119)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:114)
	at java.lang.Thread.run(Thread.java:744)

Comment by Jeffrey Yemin [ 22/Sep/14 ]

This doesn't surprise me, since all the threads are waking up at the same time. How close to the wakeup time did you restart the server?

One thing you can do is to increase the heartbeat frequency using com.mongodb.MongoClientOptions.Builder#heartbeatFrequency. It defaults to checking each server every 5 seconds, but if you bring that up to, for example, every 0.5 seconds, you'll see fewer read operation failures.

Comment by Mohamed Nouh [ 22/Sep/14 ]

Thanks Jeff,
I've made some minor changes to the code to simulate different threads. See attached.
I was able to replicate using the attached. It eventually does reconnect, but it requires that all threads error out before they can be reused, this could be problematic if I have a very large pool of connections.

Main Sleeping for 30 seconds...
T:14 Found one
T:13 Found one
T:13 Sleeping for 20 seconds
T:14 Sleeping for 20 seconds
T:15 Found one
T:15 Sleeping for 20 seconds
<Manual Mongo Server Restart>
T:14 Waking up
T:13 Waking up
T:15 Waking up
T:15 Error: Read operation to server localhost:23750 failed on database test
T:14 Error: Read operation to server localhost:23750 failed on database test
T:13 Error: Read operation to server localhost:23750 failed on database test
T:14 Sleeping for 20 seconds
T:15 Sleeping for 20 seconds
T:13 Sleeping for 20 seconds
Main Invoking Search....
T:1 Found one
T:1 Sleeping for 20 seconds
T:14 Waking up
T:13 Waking up
T:15 Waking up
T:15 Found one
T:15 Sleeping for 20 seconds
T:13 Found one
T:13 Sleeping for 20 seconds
T:14 Found one
T:14 Sleeping for 20 seconds

Comment by Jeffrey Yemin [ 20/Sep/14 ]

The short answer here is that your code has to handle occasional failure situations when servers go down. But the driver will eventually reconnect when the server comes back online. Try running this small test program to see what I mean:

    public static void main(String[] args) throws UnknownHostException, InterruptedException {
        MongoClient client = new MongoClient();
 
        DBCollection c = client.getDB("test").getCollection("test");
 
        while (true) {
            try {
                c.findOne();
                System.out.println("Found one");
            } catch (Exception e) {
                System.out.println(e.getMessage());
            }
            System.out.println("Sleeping for 10 seconds");
            Thread.sleep(10000);
            System.out.println("Waking up");
        }
    }

While it's running, shut down and restart mongod as often as you like and observe the behavior of the program as you do so.

Comment by Mohamed Nouh [ 20/Sep/14 ]

I'm connecting to a standalone, but the code passes a list of server addresses to the MongoClient. There is only one element in the list.

Comment by Jeffrey Yemin [ 20/Sep/14 ]

No, autoConnectRetry is not going to help. Are you connecting to a standalone, a replica set, or a mongos?

Comment by Mohamed Nouh [ 20/Sep/14 ]

Hi Jeff,

I do not see that exception over and over, it only appears when I invoke the search. After bringing up Mongo, I've waited 10+ minutes, but had no luck getting it to restablish connection. Only way right now seems for me to manually restart the app. Should I be setting the autoConnectRetry to true? It seems deprecated according to the documentation.

Regards,
Mohamed

Comment by Jeffrey Yemin [ 20/Sep/14 ]

Hi Mohamed,

Do you see the above exception over and over again until you restart the application? After you bring mongod back up, how long are you waiting for the application to recover before restarting it?

Comment by Mohamed Nouh [ 20/Sep/14 ]

Hi Jeff,

Sorry for the delay. See below stack trace. What I've done is essentially shutdown mongo and bring it back up, the application doesn't seem to reconnect. Restarting the application, connection is established fine.

Mongo{authority=MongoAuthority{type=Set, serverAddresses=[hostname:23750], credentials={credentials={}}}, options=MongoOptions{description='null', connectionsPerHost=1000, threadsAllowedToBlockForConnectionMultiplier=5, maxWaitTime=30000, connectTimeout=20000, socketTimeout=0, socketKeepAlive=false, autoConnectRetry=false, maxAutoConnectRetryTime=0, slaveOk=false, readPreference=primaryPreferred, dbDecoderFactory=DefaultDBDecoder.DefaultFactory, dbEncoderFactory=DefaultDBEncoder.DefaultFactory, safe=false, w=0, wtimeout=0, fsync=false, j=false, socketFactory=javax.net.DefaultSocketFactory@7a93cf18, cursorFinalizerEnabled=true, writeConcern=WriteConcern { "getlasterror" : 1} / (Continue on error? false), alwaysUseMBeans=false, requiredReplicaSetName=null}}EX:com.mongodb.MongoException$Network: Read operation to server hostname:23750 failed on database inactive<code>
com.mongodb.MongoException$Network: Read operation to server hostname:23750 failed on database inactive
        at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:300)
        at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
        at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
        at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
        at com.mongodb.DBCursor._check(DBCursor.java:458)
        at com.mongodb.DBCursor._hasNext(DBCursor.java:546)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
        at com.company.search(MongoDataStore.java:176)
....
Caused by: java.io.EOFException
        at org.bson.io.Bits.readFully(Bits.java:50)
        at org.bson.io.Bits.readFully(Bits.java:35)
        at org.bson.io.Bits.readFully(Bits.java:30)
        at com.mongodb.Response.<init>(Response.java:42)
        at com.mongodb.DBPort$1.execute(DBPort.java:141)
        at com.mongodb.DBPort$1.execute(DBPort.java:135)
        at com.mongodb.DBPort.doOperation(DBPort.java:164)
        at com.mongodb.DBPort.call(DBPort.java:135)
        at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)

Comment by Jeffrey Yemin [ 08/Sep/14 ]

Please post a full stack trace in a comment. I'd like to know exactly what you mean when you say that the driver is not auto-reconnecting. Also, please let us know if your application is connecting to a standalone, a replica set, or a mongos, and what is happening with your MongoDB servers during the time when you're experiencing a problem.

Generated at Thu Feb 08 08:54:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.