[JAVA-915] ConnectionWaitTImeout thrown after upgrading driver Created: 01/Aug/13  Updated: 02/Oct/13  Resolved: 16/Aug/13

Status: Closed
Project: Java Driver
Component/s: API
Affects Version/s: 2.11.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Fabian Gebert Assignee: Unassigned
Resolution: Duplicate Votes: 3
Labels: regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04 LTS 64-bit Java 7 Oracle JDK


Issue Links:
Duplicate
duplicates JAVA-853 Client runs out of connections with n... Closed

 Description   

Upgrading from 2.10 to 2.11 the (request and db-intense) web app fails after about two days with the following exception:

Connection wait timeout after 120000 ms. Stacktrace follows:
com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 120000 ms

No code change, only upgraded MongoDB.

Is there anything we've to change when upgrading from 2.10 to 2.11?

See other reporters at the Grails MongoDB issue http://jira.grails.org/browse/GPMONGODB-307 - this is however NOT RELATED TO THE GRAILS PLUGIN (see my comment in that issue)

Thank you!



 Comments   
Comment by Graeme Rocher [ 02/Oct/13 ]

As I mentioned in JAVA-853. This is still an issue in 2.11.3 and the root cause of http://jira.grails.org/browse/GPMONGODB-307

Only downgrading to an older version of the driver fixes the problem. Can someone please reopen and address this issue?

Comment by Noam Y. Tenne [ 16/Aug/13 ]

That's great, thanks!

Comment by Jeffrey Yemin [ 16/Aug/13 ]

Now that we know the root cause, I see this is a duplicate of JAVA-853. I'm going to resolve this one, and link it

Comment by Fabian Gebert [ 10/Aug/13 ]

I don't explicitly use mongo session, but I guess when using GORM, it will automatically be used.
So you think it can be fixed?

Comment by Jeffrey Yemin [ 10/Aug/13 ]

MongoSession uses them:

Are you using GORM sessions, by any chance? If so, I think I see the what the problem is.

Comment by Fabian Gebert [ 09/Aug/13 ]

Hi Jeff,

I'm not using GPars. And I've never heard of DB.requestStart nor DB.requestDone. Perhaps the GORM plugin does use these methods?

Comment by Jeffrey Yemin [ 09/Aug/13 ]

Hi Noam,

The only thread local state we keep in the driver is due to use of DB.requestStart and DB.requestDone, and the thread local management did change in 2.11. Does your application by any chance use these methods along with GPars? Can you share any of the code?

Fabian, are you also using GPars and or DB.requestStart/requestDone?

Comment by Noam Y. Tenne [ 07/Aug/13 ]

So I hadn't had too much time to dig in any deeper, but coming from the context of a Grails application it seems that the issue is tied to operations done in external threads.
When I upgrade the Java driver from 2.10.x to 2.11.x (changing nothing else in the application), operations done asynchronously via libraries such as GPars start to leak DBPorts; perhaps some ThreadLocal state is saved somewhere?
I might have time to build a reproducible test case later on in the week.

Comment by Noam Y. Tenne [ 04/Aug/13 ]

I too am affected by this issue

Comment by Fabian Gebert [ 01/Aug/13 ]

My config is default except for

connectionsPerHost = 200
threadsAllowedToBlockForConnectionMultiplier = 750
autoConnectRetry = true
connectTimeout = 15000

> Why are you sure this is not a change in the behavior/load on the web app?

Because we reverted back to 2.10 and it is working again under the same load.

> What is the size of the connection pool (connectionsPerHost in MongoClientOptions)?

200

> How many threads are created by the container?

We've got a Tomcat 7 in default settings. I think that's about 100 threads, am I right?

> Would it be possible to attach a thread dump from around the time of the exception?

I cannot test it on the production machine, as we don't want to break it. I can try to reproduce the error on a test environment, however.

> Does the exception keep getting thrown, or is intermittent?

It gets thrown on all subsequent requests so the server is dead (except for calls not involving database)

> Are you setting a socket timeout in MongoClientOptions?

No, we've got a nightly map reduce running, I want to avoid that to be killed. The map reduce always successfully finished so far, no matter what driver version

Hope that helps!

Comment by Jeffrey Yemin [ 01/Aug/13 ]

Some questions:

  • Why are you sure this is not a change in the behavior/load on the web app?
  • What is the size of the connection pool (connectionsPerHost in MongoClientOptions)?
  • How many threads are created by the container?
  • Would it be possible to attach a thread dump from around the time of the exception?
  • Does the exception keep getting thrown, or is intermittent?
  • Are you setting a socket timeout in MongoClientOptions?
Generated at Thu Feb 08 08:53:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.