-
Type: Bug
-
Resolution: Cannot Reproduce
-
Priority: Blocker - P1
-
None
-
Affects Version/s: 2.12.1
-
Component/s: Cluster Management
-
Labels:None
Hello,
We were impacted by the bugs resolved in version 2.12.1
(https://github.com/mongodb/mongo-java-driver/releases/tag/r2.12.1)
Thus we upgraded this morning to the new driver and after the production push we get a lot of errors coming from MongoDriver, which crashes preventing web request to be fulfilled, thus having a high rate of http status code 500 errors.
Our application is hosted on Heroku, where there were also no issue at the moment.
We are using a MongoHQ Master/Slave replication set running MongoDB 2.6.0, which run well. There were no failover that could cause the issue.
I cannot say whether Heroku platform or MongoHQ could have broke the DB connection for some reason.
On MongoHQ, the current master is candidate.6, candidate.5 being the slave. There is also another slave and an arbitrer, but they are not used in the connection string.
The connection string is:
"mongodb://<user>:<pass>@candidate.6.mongolayer.com:10151,candidate.5.mongolayer.com:10151/guestful"
In any case, here are the exceptions we get:
Caused by: com.mongodb.MongoException$Network: Read operation to server c151.candidate.6.mongolayer.com:10151 failed on database guestful at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:308) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:279) at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84) at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66) at com.mongodb.DBCollection.findOne(DBCollection.java:870) at com.mongodb.DBCollection.findOne(DBCollection.java:844) at com.mongodb.DBCollection.findOne(DBCollection.java:802) at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at com.guestful.backend.service.DB$GCollection.findOne(DB.groovy:186) at com.guestful.backend.service.DB$GCollection$findOne$12.call(Unknown Source) at com.guestful.backend.rest.service.RestaurantHelper.findOne(RestaurantHelper.groovy:61) at com.guestful.backend.rest.service.RestaurantHelper$findOne.call(Unknown Source) at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilities(RestaurantAvailabilityResource.groovy:125) at com.guestful.backend.rest.RestaurantAvailabilityResource$getAvailabilities.callCurrent(Unknown Source) at com.guestful.backend.rest.RestaurantAvailabilityResource.getAvailabilitiesForRestaurant(RestaurantAvailabilityResource.groovy:44) at sun.reflect.GeneratedMethodAccessor613.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ... 59 common frames omitted Caused by: java.io.EOFException: null at org.bson.io.Bits.readFully(Bits.java:50) at org.bson.io.Bits.readFully(Bits.java:35) at org.bson.io.Bits.readFully(Bits.java:30) at com.mongodb.Response.<init>(Response.java:42) at com.mongodb.DBPort$1.execute(DBPort.java:141) at com.mongodb.DBPort$1.execute(DBPort.java:135) at com.mongodb.DBPort.doOperation(DBPort.java:164) at com.mongodb.DBPort.call(DBPort.java:135) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:300) ... 109 common frames omitted
and
com.mongodb.MongoException$Network: Operation on server c151.candidate.6.mongolayer.com:10151 failed at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:225) at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:489) at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:428) at com.mongodb.DBCollectionImpl.updateWithCommandProtocol(DBCollectionImpl.java:415) at com.mongodb.DBCollectionImpl.update(DBCollectionImpl.java:267) at com.mongodb.DBCollection.update(DBCollection.java:191) at com.mongodb.DBCollection.update(DBCollection.java:224) at sun.reflect.GeneratedMethodAccessor491.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaMethod$doMethodInvoke.call(Unknown Source) at com.gmongo.internal.Patcher._invoke(Patcher.groovy:49) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) at com.gmongo.internal.Patcher$__patchInternal_closure1.doCall(Patcher.groovy:38) at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1108) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) at org.codehaus.groovy.runtime.callsite.PerInstancePojoMetaClassSite.call(PerInstancePojoMetaClassSite.java:35) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128) at com.guestful.backend.service.DB$GCollection.update(DB.groovy:202) at com.guestful.backend.service.DB$GCollection.update(DB.groovy) at com.guestful.backend.service.DB$GCollection$update$15.call(Unknown Source) at com.guestful.backend.event.handler.Stats.onAvailabilityCallEvent(Stats.groovy:129) at sun.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:189) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at com.ovea.tajin.framework.async.guava.EventHandler.handleEvent(EventHandler.groovy:72) at com.ovea.tajin.framework.async.guava.EventHandler$handleEvent.call(Unknown Source) at com.ovea.tajin.framework.async.guava.AsyncEventBus$1.run(AsyncEventBus.groovy:100) at java_lang_Runnable$run.call(Unknown Source) at com.ovea.tajin.framework.async.guice.AsyncModule$Task.run(AsyncModule.groovy:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.EOFException: null at org.bson.io.Bits.readFully(Bits.java:50) at org.bson.io.Bits.readFully(Bits.java:35) at org.bson.io.Bits.readFully(Bits.java:30) at com.mongodb.Response.<init>(Response.java:42) at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:492) at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:489) at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:218) ... 54 common frames omitted
There are very frequent compared to before 2.12.1 driver version, and cause requests to fail.