On Tue, Feb 19, 2013 at 7:13 PM, Alberto Vilches <vilches@gmail.com> wrote:
Hi, I'm using Mongodb to log the access in a public Grails application, using Mongodb Java client 2.10.1, and it works fine (we have up to 30 request/second, and every request is logged!).

Now, we are trying to stress the system, doing some chaos monkey tests, like restart the mongodb in the middle of the day. While the mongodb is restarting, the logs are losts, and this is the behavior expected. When the mongodb is up and running again, the app continues sending logs to the Mongodb. At the beginning, only a few are lost with a timeout, but when the time passes, more and more logs are lost and finally, no logs are send to the Mongo. This is the exception we have:

org.springframework.data.mongodb.UncategorizedMongoDbException: Connection wait timeout after 120000 ms; nested exception is com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 120000 ms

My Mongodb configuration is this:
grails.mongo.host=mongo.mundoreader.local
grails.mongo.options.autoConnectRetry=true
grails.mongo.options.connectTimeout=300
grails.mongo.options.connectionsPerHost=40
grails.mongo.options.maxAutoConnectRetryTime=5
grails.mongo.options.maxWaitTime=120000
grails.mongo.options.threadsAllowedToBlockForConnectionMultiplier=50

I'm not sure, but I have the felling the connection pool keep the old and broken connections alive, so when the Mongodb starts again, the connection pool doesn't have any connection free to connect to it again, and wait forever to the broken and old connection finish. Like the sql connection pool, the Mongo connection pool should check if a connection is broken, and remove it from the pool, creating a new one, but I think that's not happening.

¿Somebody have any idea about this? Thank you!

–
Un saludo.
Alberto Vilches
http://albertovilches.com
Twitter: @albertovilches