[JAVA-2172] When Mongo is down, Hystrix circuit breaker won't open. Created: 20/Apr/16  Updated: 11/Sep/19  Resolved: 15/Sep/16

Status: Closed
Project: Java Driver
Component/s: Async, Configuration
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Minor - P4
Reporter: Bart?omiej Szwej Assignee: Unassigned
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Hello,
I'm using Java Driver Rx (v1.2.0) with MongoDB (v3.2.1).
I'm caching Mongo clients. When the first call to db is made, the client is created and makes the call.
Then, when Mongo goes down and I try to use this cached client, all subsequent requests are "hanging" (because they're queued in the client). This behaviour causes hystrix command timeout.
Since it doesn't timeout or throw any exceptions (it just hangs), therefore the circuit breaker won't open.



 Comments   
Comment by Ross Lawley [ 20/Apr/16 ]

Hi DamianJ,

The MongoClient won't throw an exception as the connection pool is designed to be self healing across ReplicaSet reelections etc. However, when calling operations it should return an error to the callback, I'm going to try and reproduce your scenario with a quick test case for this and come back to you.

Ross

Comment by Damian Jureczko [X] [ 20/Apr/16 ]

Hi Ross,

I'm working on it with Bartek.

We've tried with these setting and with many more and we couldn't find any configuration that cause MongoClient to throw an exception or fail an operation in any way.
We've tested it with Async driver and we don't get any feedback in the callback. No result. No error. With this it's difficult to be responsive.
Another thing is that operations are queued and performed later when Mongo is up again. Might lead to inconsistency in our data.

Best Regards,
Damian

Comment by Ross Lawley [ 20/Apr/16 ]

Hi bszwej,

I think it will be down to a configuration issue of the MongoClient, as there are some timeouts incase a MongoD goes down:

These can be configured via: MongoClientSettings
Two configurations are of note:
1) SocketSettings.Builder#connectTimeout The socket connect timeout - defaults to 10,000MS
2) ClusterSettings.Builderl#serverSelectionTimeout the timeout to apply when selecting a server from the connection pool - defaults to 30,000MS

So any requests after the Java Driver will wait for these timeouts before throwing an error. Is the hystrix timeout shorter? That would account for it timing out.

For future reference this project is for Java driver bugs or feature requests and the best place for questions regarding MongoDB usage or the Java driver specifics is the mongodb-user mailing list or stackoverflow as you will reach a boarder audience there. If your business requires an answer from MongoDB within a set time frame then we do offer production support.

Ross

Comment by Bart?omiej Szwej [ 20/Apr/16 ]

In other words, we expect the driver to throw exceptions when Mongo is down, but not queue up the requests.

Generated at Thu Feb 08 08:56:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.