Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Cannot Reproduce
Priority: Blocker - P1
Fix Version/s: None
Affects Version/s: 2.12.1
Component/s: Cluster Management
Labels:
None

Hello,

We were impacted by the bugs resolved in version 2.12.1
(https://github.com/mongodb/mongo-java-driver/releases/tag/r2.12.1)

Thus we upgraded this morning to the new driver and after the production push we get a lot of errors coming from MongoDriver, which crashes preventing web request to be fulfilled, thus having a high rate of http status code 500 errors.

Our application is hosted on Heroku, where there were also no issue at the moment.

We are using a MongoHQ Master/Slave replication set running MongoDB 2.6.0, which run well. There were no failover that could cause the issue.

I cannot say whether Heroku platform or MongoHQ could have broke the DB connection for some reason.

On MongoHQ, the current master is candidate.6, candidate.5 being the slave. There is also another slave and an arbitrer, but they are not used in the connection string.

The connection string is:

"mongodb://<user>:<pass>@candidate.6.mongolayer.com:10151,candidate.5.mongolayer.com:10151/guestful"

In any case, here are the exceptions we get:

Caused by: com.mongodb.MongoException$Network: Read operation to server c151.candidate.6.mongolayer.com:10151 failed on database guestful 
    at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:308) 
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:279) 
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84) 
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66) 
    at com.mongodb.DBCollection.findOne(DBCollection.java:870) 
    at com.mongodb.DBCollection.findOne(DBCollection.java:844) 
    at com.mongodb.DBCollection.findOne(DBCollection.java:802) 
    at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) 
    at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) 
    at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) 
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) 
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) 
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) 
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) 
    at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) 
    at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) 
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) 
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) 
    at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) 
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) 
    at com.guestful.backend.service.DB$GCollection.findOne(DB.groovy:186) 
    at com.guestful.backend.service.DB$GCollection$findOne$12.call(Unknown Source) 
    at com.guestful.backend.rest.service.RestaurantHelper.findOne(RestaurantHelper.groovy:61) 
    at com.guestful.backend.rest.service.RestaurantHelper$findOne.call(Unknown Source) 
    at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilities(RestaurantAvailabilityResource.groovy:125) 
    at com.guestful.backend.rest.RestaurantAvailabilityResource$getAvailabilities.callCurrent(Unknown Source) 
    at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilitiesForRestaurant(RestaurantAvailabilityResource.groovy:44) 
    at sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) 
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) 
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) 
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) 
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) 
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) 
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) 
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) 
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) 
    	... 59 common frames omitted 
Caused by: java.io.EOFException: null 
    at org.bson.io.Bits.readFully(Bits.java:50) 
    at org.bson.io.Bits.readFully(Bits.java:35) 
    at org.bson.io.Bits.readFully(Bits.java:30) 
    at com.mongodb.Response.<init>(Response.java:42) 
    at com.mongodb.DBPort$1.execute(DBPort.java:141) 
    at com.mongodb.DBPort$1.execute(DBPort.java:135) 
    at com.mongodb.DBPort.doOperation(DBPort.java:164) 
    at com.mongodb.DBPort.call(DBPort.java:135) 
    at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:300) 
    	... 109 common frames omitted

and

com.mongodb.MongoException$Network: Operation on server c151.candidate.6.mongolayer.com:10151 failed 
    at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:225) 
    at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:489) 
    at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:428) 
    at com.mongodb.DBCollectionImpl.updateWithCommandProtocol(DBCollectionImpl.java:415) 
    at com.mongodb.DBCollectionImpl.update(DBCollectionImpl.java:267) 
    at com.mongodb.DBCollection.update(DBCollection.java:191) 
    at com.mongodb.DBCollection.update(DBCollection.java:224) 
    at sun.reflect.GeneratedMethodAccessor491.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) 
    at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) 
    at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) 
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) 
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) 
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) 
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) 
    at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) 
    at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) 
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) 
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) 
    at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) 
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128) 
    at com.guestful.backend.service.DB$GCollection.update(DB.groovy:202) 
    at com.guestful.backend.service.DB$GCollection.update(DB.groovy) 
    at com.guestful.backend.service.DB$GCollection$update$15.call(Unknown Source) 
    at com.guestful.backend.event.handler.Stats.onAvailabilityCallEvent(Stats.groovy:129) 
    at sun.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:491) 
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189) 
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) 
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) 
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55) 
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) 
    at com.ovea.tajin.framework.async.guava.EventHandler.handleEvent(EventHandler.groovy:72) 
    at com.ovea.tajin.framework.async.guava.EventHandler$handleEvent.call(Unknown Source) 
    at com.ovea.tajin.framework.async.guava.AsyncEventBus$1.run(AsyncEventBus.groovy:100) 
    at java_lang_Runnable$run.call(Unknown Source) 
    at com.ovea.tajin.framework.async.guice.AsyncModule$Task.run(AsyncModule.groovy:154) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:724) 
Caused by: java.io.EOFException: null 
    at org.bson.io.Bits.readFully(Bits.java:50) 
    at org.bson.io.Bits.readFully(Bits.java:35) 
    at org.bson.io.Bits.readFully(Bits.java:30) 
    at com.mongodb.Response.<init>(Response.java:42) 
    at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:492) 
    at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:489) 
    at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:218) 
    	... 54 common frames omitted

There are very frequent compared to before 2.12.1 driver version, and cause requests to fail.

Assignee:: Unassigned

Reporter:: Mathieu Carbou

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: Apr 30 2014 05:16:39 PM UTC

Updated:: Jun 25 2014 06:05:08 PM UTC

Resolved:: Apr 30 2014 08:21:18 PM UTC

Details

Description

Attachments

Activity

People

Dates