[JAVA-3578] Avoiding MongoWaitQueueFullExceptions by overriding ObservableImplicits Created: 04/Dec/19  Updated: 28/Oct/23  Resolved: 30/Jan/20

Status: Closed
Project: Java Driver
Component/s: Scala
Affects Version/s: None
Fix Version/s: 4.0.0

Type: Improvement Priority: Major - P3
Reporter: Colin Fairless Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File Main.scala    
Issue Links:
Related
Case:
Backwards Compatibility: Minor Change

 Description   

We would like to apply some throttling on mongo queries to avoid hitting MongoWaitQueueFullExceptions at runtime. We would also like to apply this non-invasively to our current mongo api usage.

We have managed to do this by providing alternative implicit conversions, in the same way as those in `org.mongodb.scala.ObservableImplicits`, where our version of `toFuture` (and `toFutureOption`) would evaluate the `observable: Observable` on a fixed-size thread-pool.

The observable, which the extra methods are added to, needs to be captured as a by-name parameter, in-order that we can evaluate it on our thread-pool, e.g. `implicit class ScalaObservable[T](observable: => Observable[T])`.

But since an implicit for a by-name parameter is picked up secondary to a by-value parameter, we are unable to override the predefined `toFuture` function, and have to create a new function (e.g. `toFutureThrottled`).

If the provided implicit classes in `ObservableImplicits`:

  `implicit class ScalaObservable[T](observable: Observable[T])`

were changed to:
```

implicit class ScalaObservable[T](observable: => Observable[T]) {
  val obs = observable
 …

```

(and ScalaSingleObservable similarly), then we could override them, such that `toFuture` behaves as we intend.

Is this worth a pull-request, to change the observable parameter from a by-value to a by-name parameter?

Or is there an alternative recommendation for achieving this goal?



 Comments   
Comment by Githook User [ 30/Jan/20 ]

Author:

{'username': 'rozza', 'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com'}

Message: Scala: Use pass by name in implicits functions

Allows the functions to be overridden by users

JAVA-3578
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/b8d844d3f9871a5c62045db2b483847ec0eec876

Comment by Ross Lawley [ 27/Jan/20 ]

PR: https://github.com/rozza/mongo-java-driver/pull/355

I think this change will be safe to backport to Scala 2.9.0

Comment by Colin Fairless [ 17/Jan/20 ]

Hi Ross Lawley,

I think it's a case of naming the override the same as the one to be overriden.

I have uploaded a simplified example.Main.scala

Comment by Ross Lawley [ 13/Jan/20 ]

Hi colin.fairless@digital.hmrc.gov.uk,

Apologies, I've not been able to override the implicit ScalaObservable toFuture method without causing an ambiguity error: "implicit conversions are not applicable because they are ambiguous".

Could you provide an example of how you'd want to extend the implicit?

Kind regards,

Ross

Comment by Ross Lawley [ 08/Jan/20 ]

Hi colin.fairless@digital.hmrc.gov.uk,

Just to update this is now scheduled for the 4.0.0 release, which is now the next planned release of the scala driver. Why a 4.0 release and not a 3.0 release? This is because the Scala code base has now been merged into Java driver repo ensuring the API's always stay aligned as development goes forward. The 3.12 release of the Java driver which is used by the 2.8 Scala driver release, is the last planned release in the 3.x series.

As these implicits are used through out userland code changing the implementation of them in a minor release maybe breaking the semver contract and as such changing in a major release seems more appropriate. So unfortunately, until the 4.0 release is available, you'll have to continue to use your customised implicits.

All the best,

Ross

Comment by Ross Lawley [ 16/Dec/19 ]

Thanks for the feedback colin.fairless@digital.hmrc.gov.uk,

I've scheduled this for the next release (2.9.0) which should be released early 2020.

Ross

Comment by Colin Fairless [ 16/Dec/19 ]

Hi Ross Lawley,

We have looked at increasing the wait pool size, although under load this can still be reached. We previously used reactive-mongo, which uses akka to throttle load to the database, and for backward compatibility want to preserve this behaviour.

The observeOn helper does not really help here, since, for the same reason, it is applied after evaluating the Observer.

Comment by Ross Lawley [ 10/Dec/19 ]

Hi colin.fairless@digital.hmrc.gov.uk,

Thanks for the ticket , for future reference the best place for questions would be the mongodb-user mailinglist or stackoverflow as you will reach a boarder audience there. If your business requires an answer from MongoDB within a time frame then we do offer production support which has a different avenue for support questions.

In general it may be worth looking at configuring the wait pool size and increase it via the MongoClientSettings - is that something you have already tried?
You certainly can throttle at a higher level and there is the observeOn helper to set the execution context for the Observable, which you may already be using.

I'd need to consider the implications of the change to by value and see if there are any negatives before proceeding with the change.

Ross

Generated at Thu Feb 08 08:59:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.