[JAVA-581] Background threads continue to run after a web app is undeployed unless Mongo#close is called Created: 05/Jun/12  Updated: 25/Jun/13  Resolved: 25/Jun/13

Status: Closed
Project: Java Driver
Component/s: Cluster Management
Affects Version/s: 2.8.0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Emmanuel VINET Assignee: Jeffrey Yemin
Resolution: Won't Fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 3.1.1 / Mongo 2.0.4 / Replicat Set 2 servers + one arbiter


Issue Links:
Duplicate
is duplicated by JAVA-636 Massive amounts of Null Pointer Excep... Closed
is duplicated by JAVA-796 With redeploy web app started a threa... Closed

 Description   

When reloading the same or a new version of an ejb using the mongo driver 2.8.0, NullPointerException occurs continuously in Glassfish logs as the precedent thread was not stopped and tries too use a closed connection.
We need to restart Glassfish server each time we want to redeploy our code, which is not possible.
This didn't happen with the 2.7.3 version of the driver.

[#|2012-06-05T15:18:39.418+0200|WARNING|glassfish3.1.1|com.mongodb.ReplicaSetStatus.wolverine|_ThreadID=24;_ThreadName=Thread-3;|Server seen down: localhost/127.0.0.1:27018
java.lang.NullPointerException
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.OutMessage.query(OutMessage.java:34)
at com.mongodb.DBPort.findOne(DBPort.java:144)
at com.mongodb.DBPort.runCommand(DBPort.java:156)
at com.mongodb.ReplicaSetStatus$UpdatableNode.update(ReplicaSetStatus.java:507)
at com.mongodb.ReplicaSetStatus$Updater.updateAll(ReplicaSetStatus.java:708)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:674)

#]

[#|2012-06-05T15:18:39.494+0200|WARNING|glassfish3.1.1|com.mongodb.ReplicaSetStatus.wolverine|_ThreadID=24;_ThreadName=Thread-3;|Server seen down: localhost/127.0.0.1:27017
java.lang.NullPointerException
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.OutMessage.query(OutMessage.java:34)
at com.mongodb.DBPort.findOne(DBPort.java:144)
at com.mongodb.DBPort.runCommand(DBPort.java:156)
at com.mongodb.ReplicaSetStatus$UpdatableNode.update(ReplicaSetStatus.java:507)
at com.mongodb.ReplicaSetStatus$Updater.updateAll(ReplicaSetStatus.java:708)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:674)

#]


 Comments   
Comment by Jeffrey Yemin [ 25/Jun/13 ]

The driver has to create threads to monitor server health, and the only way to stop the threads is to call Mongo.close(). I don't see how we can change this.

Comment by wanninja [ 22/Oct/12 ]

Hello,

I have same problem with Glassfish 3.1.2 and driver 2.9.1.
Anyone have a solution ?

thx

Comment by Bob Nope [ 02/Oct/12 ]

I found some strange behavior. Using a different Mongo constructor prevents the endless ReplicaSetStatus exceptions. No explicit calls to Mongo.close() were made.

 
   public @Bean
   MongoDbFactory mongoDbFactory() throws Exception
   {
 .
 . Pull servers and ports from properties file...
 .
     if (addrs.size() > 1)
       mongo = new Mongo(addrs);
     else
       mongo = new Mongo(addrs.get(0).getHost(), addrs.get(0).getPort()); 
 
     return new SimpleMongoDbFactory(mongo, SCOUT_DB);
   }

This stackoverflow question clued me in:

http://stackoverflow.com/questions/5753981/endless-mongodb-replicasetstatus-updater-exceptions-using-the-java-driver-in-spr

Comment by Bob Nope [ 02/Oct/12 ]

I am also using 2.8.0. Server: Apache Tomcat 7. The jar is part of the WAR. I have the same problem when I do a partial deploy. I thought you should be aware that the spring-data-mongodb documentation makes no mention of the need to call close. I am using the annotation based configuration.

http://static.springsource.org/spring-data/data-mongodb/docs/current/reference/html/#mongodb-connectors

I'm also not sure how/where I should call close. I tried adding a @PreDestory method w/ mongo.close() to the @Configuration class but that did not solve the problem.

Comment by Thomas Pham [ 17/Sep/12 ]

if you add the driver into glassfish/domains/

{your_domain}

/lib/ext you don't have the problem anymore (have been test with 2.9.1)

Comment by Jeffrey Yemin [ 05/Sep/12 ]

Not sure what we can do about this besides better documentation.

Comment by Jeffrey Yemin [ 03/Sep/12 ]

Hi James,

That change you made causes the driver to create a background thread that monitors the state of the replica set, so calling Mongo#close when the context is destroyed will be important for your application.

Comment by James Green [ 03/Sep/12 ]

FWIW I've just started seeing this, but my pom.xml imports mongo-java-driver:2.7.3. In my case I'm running some integration tests against Glassfish 3.1, each of which uses Arquillian. As of today, the build itself works but Glassfish is issuing this messages which I've not been before.

I made one change late last week in my code: When constructing a new Mongo(), rather than connect using the default empty constructor I'm now passing in a list of ServerAddress objects. The list consists of one ServerAddress object, using it's default constructor.

This might be a red herring of course...

Comment by Jeffrey Yemin [ 06/Jun/12 ]

Hi Emmanuel,

We're able to reproduce this problem is Glassfish. The Glassfish server intentionally nulls out all static fields in classes when the web application is undeployed, and that, combined with the lack of a call to Mongo#close, is what's causing these errors.

We're going to see what we can do to work around this in the driver, but for now I recommend you call Mongo#close in your application. When we do that in our test web app, the error messages go away.

Comment by Ryan Nitz [ 06/Jun/12 ]

Hi Emmanuel,

Can you tell us if mongo.jar is in the system classpath or in the war/ear for the app?

Also, are you deploying a completely new app (i.e., full hot deploy) or are you simply replacing a single bean?

Thanks,

-Ryan

Comment by Jeffrey Yemin [ 06/Jun/12 ]

Can you try calling Mongo#close when the context is destroyed? See http://stackoverflow.com/questions/1762246/how-to-tie-the-lifecycle-for-a-spring-bean-to-the-webapps-lifecycle for some examples of ways to do this with Spring.

Comment by Emmanuel VINET [ 06/Jun/12 ]

We are using mongo java driver through spring. Insertion are constant and we don't use close.
This problem happen when we redeploy a bean connecting Mongo, so I'm not able to control the connexion.

Comment by Jeffrey Yemin [ 05/Jun/12 ]

First of all, thank you for taking the time to test the release candidate. I have a couple of questions:

  1. Are you calling Mongo#close, and if so, when?
  2. Are you anywhere in your code changing the value of the static field
    com.mongodb.DefaultDBEncoder#FACTORY? Looking at the OutMessage
    source, that seems to be where the NPE is coming from:
    https://github.com/mongodb/mongo-java-driver/blob/a5fae2c287d07c7669a7575821ce836a998db200/src/main/com/mongodb/OutMessage.java#L38

The main difference in the behavior of close() is that in 2.7.3 a (unsynchronized) boolean flag is set (and then checked in the run
loop), while in 2.8.0 it calls Thread#interrupt (and Thread#isInterrupted is checked in the run loop). I'm wondering if that
idiom doesn't interact well with Glassfish.

Generated at Thu Feb 08 08:52:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.