[JAVA-3578] Avoiding MongoWaitQueueFullExceptions by overriding ObservableImplicits Created: 04/Dec/19 Updated: 28/Oct/23 Resolved: 30/Jan/20 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | Scala |
| Affects Version/s: | None |
| Fix Version/s: | 4.0.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Colin Fairless | Assignee: | Ross Lawley |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Case: | (copied to CRM) | ||||
| Backwards Compatibility: | Minor Change | ||||
| Description |
|
We would like to apply some throttling on mongo queries to avoid hitting MongoWaitQueueFullExceptions at runtime. We would also like to apply this non-invasively to our current mongo api usage. We have managed to do this by providing alternative implicit conversions, in the same way as those in `org.mongodb.scala.ObservableImplicits`, where our version of `toFuture` (and `toFutureOption`) would evaluate the `observable: Observable` on a fixed-size thread-pool. The observable, which the extra methods are added to, needs to be captured as a by-name parameter, in-order that we can evaluate it on our thread-pool, e.g. `implicit class ScalaObservable[T](observable: => Observable[T])`. But since an implicit for a by-name parameter is picked up secondary to a by-value parameter, we are unable to override the predefined `toFuture` function, and have to create a new function (e.g. `toFutureThrottled`). If the provided implicit classes in `ObservableImplicits`: `implicit class ScalaObservable[T](observable: Observable[T])` were changed to:
``` (and ScalaSingleObservable similarly), then we could override them, such that `toFuture` behaves as we intend. Is this worth a pull-request, to change the observable parameter from a by-value to a by-name parameter? Or is there an alternative recommendation for achieving this goal? |
| Comments |
| Comment by Githook User [ 30/Jan/20 ] |
|
Author: {'username': 'rozza', 'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com'}Message: Scala: Use pass by name in implicits functions Allows the functions to be overridden by users
|
| Comment by Ross Lawley [ 27/Jan/20 ] |
|
PR: https://github.com/rozza/mongo-java-driver/pull/355 I think this change will be safe to backport to Scala 2.9.0 |
| Comment by Colin Fairless [ 17/Jan/20 ] |
|
Hi Ross Lawley, I think it's a case of naming the override the same as the one to be overriden. I have uploaded a simplified example.Main.scala |
| Comment by Ross Lawley [ 13/Jan/20 ] |
|
Hi colin.fairless@digital.hmrc.gov.uk, Apologies, I've not been able to override the implicit ScalaObservable toFuture method without causing an ambiguity error: "implicit conversions are not applicable because they are ambiguous". Could you provide an example of how you'd want to extend the implicit? Kind regards, Ross |
| Comment by Ross Lawley [ 08/Jan/20 ] |
|
Hi colin.fairless@digital.hmrc.gov.uk, Just to update this is now scheduled for the 4.0.0 release, which is now the next planned release of the scala driver. Why a 4.0 release and not a 3.0 release? This is because the Scala code base has now been merged into Java driver repo ensuring the API's always stay aligned as development goes forward. The 3.12 release of the Java driver which is used by the 2.8 Scala driver release, is the last planned release in the 3.x series. As these implicits are used through out userland code changing the implementation of them in a minor release maybe breaking the semver contract and as such changing in a major release seems more appropriate. So unfortunately, until the 4.0 release is available, you'll have to continue to use your customised implicits. All the best, Ross |
| Comment by Ross Lawley [ 16/Dec/19 ] |
|
Thanks for the feedback colin.fairless@digital.hmrc.gov.uk, I've scheduled this for the next release (2.9.0) which should be released early 2020. Ross |
| Comment by Colin Fairless [ 16/Dec/19 ] |
|
Hi Ross Lawley, We have looked at increasing the wait pool size, although under load this can still be reached. We previously used reactive-mongo, which uses akka to throttle load to the database, and for backward compatibility want to preserve this behaviour. The observeOn helper does not really help here, since, for the same reason, it is applied after evaluating the Observer. |
| Comment by Ross Lawley [ 10/Dec/19 ] |
|
Hi colin.fairless@digital.hmrc.gov.uk, Thanks for the ticket , for future reference the best place for questions would be the mongodb-user mailinglist or stackoverflow as you will reach a boarder audience there. If your business requires an answer from MongoDB within a time frame then we do offer production support which has a different avenue for support questions. In general it may be worth looking at configuring the wait pool size and increase it via the MongoClientSettings - is that something you have already tried? I'd need to consider the implications of the change to by value and see if there are any negatives before proceeding with the change. Ross |