[JAVA-767] connection pool may leak connections on server restart Created: 21/Feb/13 Updated: 31/Mar/15 Resolved: 25/Jun/13 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Connection Management |
| Affects Version/s: | 2.10.1 |
| Fix Version/s: | 3.0.0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Jeffrey Yemin | Assignee: | Jeffrey Yemin |
| Resolution: | Done | Votes: | 4 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Description |
|
On Tue, Feb 19, 2013 at 7:13 PM, Alberto Vilches <vilches@gmail.com> wrote: Now, we are trying to stress the system, doing some chaos monkey tests, like restart the mongodb in the middle of the day. While the mongodb is restarting, the logs are losts, and this is the behavior expected. When the mongodb is up and running again, the app continues sending logs to the Mongodb. At the beginning, only a few are lost with a timeout, but when the time passes, more and more logs are lost and finally, no logs are send to the Mongo. This is the exception we have: org.springframework.data.mongodb.UncategorizedMongoDbException: Connection wait timeout after 120000 ms; nested exception is com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 120000 ms My Mongodb configuration is this: I'm not sure, but I have the felling the connection pool keep the old and broken connections alive, so when the Mongodb starts again, the connection pool doesn't have any connection free to connect to it again, and wait forever to the broken and old connection finish. Like the sql connection pool, the Mongo connection pool should check if a connection is broken, and remove it from the pool, creating a new one, but I think that's not happening. ¿Somebody have any idea about this? Thank you! – |
| Comments |
| Comment by Jeffrey Yemin [ 31/Mar/15 ] |
|
Closing all resolved 3.0.0 issues, as 3.0.0 has been tagged and released. |
| Comment by Jeffrey Yemin [ 02/Aug/13 ] |
|
h0nig Which idea were you referring to? The problem as I see it is that there is always a chance that the connection pool will return a broken connection. If we have a background thread that sends a ping to every connection every minute, you could still get a bad connection. Whatever we do only decreases the chance of it happening. By using maxLifeTime and maxIdleTime to control connection life times, we put it into the hands of the developer, who knows the characteristics of the networking environment in which the application is running, e.g. load balancers that shut down connections, etc. |
| Comment by Fabian Gebert [ 01/Aug/13 ] |
|
jeff.yemin done so with https://jira.mongodb.org/browse/JAVA-915 |
| Comment by Hans-Joachim Kliemeck [ 01/Aug/13 ] |
|
jeff.yemin thats a good idea, but there is still the chance that the connection pool will return a broken connection and will close it after the socketexception is thrown (as far as i understood your comment & code). with your change you just minimized the problem to the first connection and the fact that the other connections will be closed after the error is detected. there is still the idea of a background thread that checks the connections (and maybe handles the max* attributes) |
| Comment by Jeffrey Yemin [ 01/Aug/13 ] |
|
h0nig I prefer specifying maxIdleTime and maxLifeTime, as discussed in |
| Comment by Jeffrey Yemin [ 01/Aug/13 ] |
|
fabiangebert This bug was reported against the 2.10 version of the driver, so it's not likely to be what your encountering there, as I understand it you believe the problem in GPMONGO-307 as a regression introduced in 2.11. Would you mind opening a new issue related to your findings in GPMONGODB-307? |
| Comment by Fabian Gebert [ 01/Aug/13 ] |
|
is this related to http://jira.grails.org/browse/GPMONGODB-307 ? |
| Comment by Hans-Joachim Kliemeck [ 26/Jun/13 ] |
|
do you think that is also a good idea to have a thread running on the background that is checking the connections periodically? i think yes |
| Comment by Jeffrey Yemin [ 25/Jun/13 ] |
|
See In PooledConnection, the wrapped connections is discarded if it's been closed, which will be the case if there was a socket exception. |
| Comment by Jeffrey Yemin [ 25/Jun/13 ] |
|
In 3.0.x branch, closed connections (which happens on IOException) are discarded. |
| Comment by Hans-Joachim Kliemeck [ 02/May/13 ] |
|
got the same problem. there should be a precheck on returning the port from the pool. https://github.com/mongodb/mongo-java-driver/pull/113 |