[JAVA-1196] Lot of Read Operation Failed after upgraded to driver 2.12.1 Created: 30/Apr/14 Updated: 25/Jun/14 Resolved: 30/Apr/14 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Cluster Management |
| Affects Version/s: | 2.12.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Mathieu Carbou | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
Hello, We were impacted by the bugs resolved in version 2.12.1 Thus we upgraded this morning to the new driver and after the production push we get a lot of errors coming from MongoDriver, which crashes preventing web request to be fulfilled, thus having a high rate of http status code 500 errors. Our application is hosted on Heroku, where there were also no issue at the moment. We are using a MongoHQ Master/Slave replication set running MongoDB 2.6.0, which run well. There were no failover that could cause the issue. I cannot say whether Heroku platform or MongoHQ could have broke the DB connection for some reason. On MongoHQ, the current master is candidate.6, candidate.5 being the slave. There is also another slave and an arbitrer, but they are not used in the connection string. The connection string is:
In any case, here are the exceptions we get:
and
There are very frequent compared to before 2.12.1 driver version, and cause requests to fail. |
| Comments |
| Comment by Mathieu Carbou [ 25/Jun/14 ] | |||||||||||||||||||||||||||||||
|
PVI - in our case, this issue was caused when using Mongo versio 2.6.0: the process kept crashing with a segfault. | |||||||||||||||||||||||||||||||
| Comment by Jeffrey Yemin [ 25/Jun/14 ] | |||||||||||||||||||||||||||||||
|
mnorman please let us know if you can provide any more details, especially application logs. | |||||||||||||||||||||||||||||||
| Comment by Michael D. Norman [ 25/Jun/14 ] | |||||||||||||||||||||||||||||||
|
We noticed this issue as well, with 2.12.0 and 2.12.2. We are downgrading to 2.11.3 to see if it fixes the issue. We are completely in AWS with our own setup (no 3rd party involved), and we weren't seeing this issue before the upgrade to 2.12.0. | |||||||||||||||||||||||||||||||
| Comment by Mathieu Carbou [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Ok thank you! | |||||||||||||||||||||||||||||||
| Comment by Jeffrey Yemin [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Hi Mathieu, OK, I will close this but don't hesitate to open a new issue if you again begin to suspect the driver. | |||||||||||||||||||||||||||||||
| Comment by Mathieu Carbou [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Note: I logged this issue because the problems occured just after the migration to the new driver version, but I strongly fear that the issue was much more on MongoHQ side. I contacted their support a while ago, left two mesages but had no response. Perhaps you could just close this issue because I cannot confirm it... If we were to have again a lot of these excetions while MongoHQ is working fine, I will TRACE The mongo driver package and send you the logs. Thank you for the fast feedback ! | |||||||||||||||||||||||||||||||
| Comment by Mathieu Carbou [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Sadly I do not have MongoHQ detailed logs and to be able to TRACE the "com.mongodb" package I would need another production deployment. I checked deeper on the stepdown alerts on our MongoHQ console and I saw a bunch of master / slave changes:
What is strange is that we were emailed about those stepdowns, but only just one, Do you think those EOF issues could be caused by the master shutdown and the slave taking its place ? I see a lot of coïncidence between the exception trace times and the stepdown logs from MongoHQ. | |||||||||||||||||||||||||||||||
| Comment by Jeffrey Yemin [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Mathieu, By the way, if you are unable to share the logs in this public forum we can provide a private means for that. Please let us know. Thanks, | |||||||||||||||||||||||||||||||
| Comment by Jeffrey Yemin [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Can you provide Java application logs for anything prefixed with "com.mongodb"? If you can get access to server logs for c151.candidate.6.mongolayer.com:10151, we'd like to see those too. The EOFException with that stack trace indicates that the call to SocketInputStream.read returned -1, which indicates a failed attempt to read the reply from the socket. | |||||||||||||||||||||||||||||||
| Comment by Jeffrey Yemin [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
Can you roll back while we gather more information? | |||||||||||||||||||||||||||||||
| Comment by Mathieu Carbou [ 30/Apr/14 ] | |||||||||||||||||||||||||||||||
|
We are using this Java code to setup the connections:
We do not use any special connection string parameter. |