[JAVA-1931] com.mongodb.MongoSocketWriteException when using Async driver + Netty Created: 24/Aug/15 Updated: 27/Jun/17 Resolved: 13/Apr/16 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Async |
| Affects Version/s: | 3.0.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Frederico Ferro Schuh | Assignee: | Ross Lawley |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows 7, application built on Vertx 3 + Netty 4.0.28 |
||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Description |
|
I'm using the new Mongo RxJava async driver, and I'm running into issues when I run it through Netty. When my application idles for about 1 minute without doing any database operation, in the next operation I'll get the following Exception:
By looking at the stack trace you can't tell exactly which operation caused the error because of the asynchronous Observables, but the above Exception happened on MongoCollection.replaceOne() operation. As this is a very young driver, I understand there may still be bugs, and I'd be glad if this stack trace could be of any use to identify them. |
| Comments |
| Comment by Frederico Ferro Schuh [ 13/Oct/15 ] | |||||||
|
Thanks Ross! Updated and working great! | |||||||
| Comment by Ross Lawley [ 08/Oct/15 ] | |||||||
|
Hi, just to let you know the rx driver 1.1 was released today! | |||||||
| Comment by Ross Lawley [ 22/Sep/15 ] | |||||||
|
Great stuff, I'm closing this ticket now! Thanks again for your help and contribution - happy MongoDB'ing! Ross | |||||||
| Comment by Frederico Ferro Schuh [ 22/Sep/15 ] | |||||||
|
Hi Ross, I'm also very happy to say I can no longer reproduce both issues on the latest snapshot of the async 3.1 and Rx 1.1 drivers. Wish you and the rest of the Mongo team the best. | |||||||
| Comment by Ross Lawley [ 21/Sep/15 ] | |||||||
|
I'm happy to say the snapshot of mongodb-driver-async has been published and I can no longer reproduce the issue. fschuh Many thanks for your patience and help diagnosing this. | |||||||
| Comment by Frederico Ferro Schuh [ 21/Sep/15 ] | |||||||
|
Great, glad to know you've found the issue! | |||||||
| Comment by Ross Lawley [ 21/Sep/15 ] | |||||||
|
OK I think I have found the cause - the new Observable abstraction in 3.1 had a bug in SingleResultCallbackSubscription}}s. It wrongly defaulted the {{completed state. Due to how the data is queued and processed multiple calls to request mean there was a race condition and the completed flag is checked and the Observable is marked as completed before data is returned. In RxJava 1.0.12 the data was only requested once - hence no bug and it working fine. There was a change in 1.0.13 with regards to how Merge works and that surfaced the bug. I have opened | |||||||
| Comment by Ross Lawley [ 21/Sep/15 ] | |||||||
|
Great work! I'll update to the latest RxJava and investigate now. | |||||||
| Comment by Frederico Ferro Schuh [ 21/Sep/15 ] | |||||||
|
Hi Ross, I found out why you can't reproduce the issue but it still happens with me. Please have a check with the newer RxJava versions, something must have changed on it that is causing the Rx driver to break. If you run your QuickStart class with the following dependencies, you can reproduce the issue:
| |||||||
| Comment by Ross Lawley [ 18/Sep/15 ] | |||||||
|
Hi, Great that the first issue has been fixed correctly. Unfortunately, I'm still am unable to duplicate the second issue with the following dependencies:
The code I'm running is here: https://github.com/rozza/mongo-java-quickstart/tree/3.0-rx-vert-test - can you confirm that you can cause it to fail when calling:
Ross | |||||||
| Comment by Frederico Ferro Schuh [ 18/Sep/15 ] | |||||||
|
Hello Ross, I have tested again, and can confirm the main1 issue no longer occurs with the latest 3.1 SNAPSHOT, as well as the latest release 3.0.4, it appears to have been fixed now. The main2 issue is still present on the latest RX 1.1 SNAPSHOT driver. | |||||||
| Comment by Ross Lawley [ 09/Sep/15 ] | |||||||
|
Hi fschuh, Just to let you know a new snapshot has been released with the fix. When testing and refreshing dependencies I'm seeing both cases work as expected. My test code is here: https://github.com/rozza/mongo-java-quickstart/tree/3.0-rx-vert-test (just so gradle can manage the dependencies). I still can't replicate the failure for the main2 (I test this just by renaming the methods in the test code posted above). Could you retest and confirm main is fixed for you? Also, if you can provide any further information on the main2 case then that would be really helpful. Ross | |||||||
| Comment by Ross Lawley [ 08/Sep/15 ] | |||||||
|
Thanks fschuh, looks like I missed an important latency based failure case in I've reopened that ticket and have a fix in review and I'll post an update this ticket once that is done. I'm still struggling to reproduce main2 failure but I'll investigate once a new mongodb-async-driver snapshot is released. Ross | |||||||
| Comment by Frederico Ferro Schuh [ 08/Sep/15 ] | |||||||
|
Hi Ross, I just tried with 3.1 SNAPSHOT dated from 2015/09/07, and I still get the main2 issue. The main1 issue is still present as well. | |||||||
| Comment by Ross Lawley [ 01/Sep/15 ] | |||||||
|
Hi @fschuh, Can you still reproduce the main2 issue? Ross | |||||||
| Comment by Ross Lawley [ 26/Aug/15 ] | |||||||
|
Hi Frederico, I renamed main2 to main and always got result: true. I'll leave this ticket open for now and review post Thanks for taking the time to report and help reproduce these issues, its appreciated! As is any further feedback on the RxJava driver. Ross | |||||||
| Comment by Frederico Ferro Schuh [ 26/Aug/15 ] | |||||||
|
Make sure you just run main2, don't run the original main as well. | |||||||
| Comment by Ross Lawley [ 25/Aug/15 ] | |||||||
|
I was unable to reproduce the issue with the latest snapshot and main2, I ran it over 50 times. So leaving that for now. | |||||||
| Comment by Frederico Ferro Schuh [ 25/Aug/15 ] | |||||||
|
Thanks Ross, I'll disable read timeout for now. Setting the timeouts on my Observables will work good as a temporary workaround. | |||||||
| Comment by Ross Lawley [ 25/Aug/15 ] | |||||||
|
Hi fschuh, I have found the issue with the NettyStream - currently the read timeout doesn't work the same as it does with the nio2 and is more akin to the MaxIdleTime setting. I've added Currently, for your code I would recommend removing the read timeout and settings the MaxIdleTime setting in ConnectionPoolSettings, that will handle the connection pool resource usage. Then with any reads / operations with MongoDB add timeout to the observable. I'm now going to look into the 3.1.0-SNAPSHOT issues. Ross | |||||||
| Comment by Ross Lawley [ 25/Aug/15 ] | |||||||
|
Thanks, I'm trying to repo and debug this now. | |||||||
| Comment by Frederico Ferro Schuh [ 25/Aug/15 ] | |||||||
|
Hi Ross, thanks for the quick follow up. And after digging a bit, I finally found the reason of the Exception. I've attached a file with a 100% reproduction test. I failed to reproduce the issue if Vertx 3 is not running. Also, the attached code contains a main2() method that performs a test that fails on 3.1.0-SNAPSHOT and Rx driver 1.1.0-SNAPSHOT, while it works correctly under 3.0.2 and Rx driver 1.0.1. The Vertx dependency you need in order to run the code (Gradle) | |||||||
| Comment by Ross Lawley [ 24/Aug/15 ] | |||||||
|
Hi fschuh, Thanks for the ticket, I see you are using Vert.x as well, can you produce any sample code that reproduces this issue? Also, could you test with the 1.1.0-SNAPSHOT? It relies on the 3.1.0-SNAPSHOT MongoDB Java Driver which contains some improvements and fixes to the underlying async driver. Ross |