[JAVA-1884] Allow limiting the amount of concurrent queries executed in the async driver Created: 10/Jul/15  Updated: 06/Dec/17  Resolved: 06/Dec/17

Status: Closed
Project: Java Driver
Component/s: Async
Affects Version/s: None
Fix Version/s: 3.6.0

Type: Improvement Priority: Major - P3
Reporter: Peter Hendriks Assignee: Ross Lawley
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

When using the async driver, we have found we cannot limit the amount of concurrent queries executed in the async execution model.

Although we'd assumed limiting the amount of connections in the connection pool would limit the amount of queries possible in flight, it turns out, this is not how it works out all the time. When the query returns a larger amount of results than the batch size (20 by default), the retrieval of results in the async model will yield the connection back to the pool after each batch has executed. In this case, an additional query request may start, and the number of concurrent queries may become larger than the connection pool.

We need a model to limit the amount of concurrent queries possible to prevent memory issues when too many queries would be processed at once. This a common use case, it would be very useful to support directly in the mongo async driver.



 Comments   
Comment by Ross Lawley [ 06/Dec/17 ]

I'm closing this as "Won't Fix" as limiting the threadpool size by providing your own Executor service as per JAVA-2561 is now the supported mechanisim to handle memory usage.

There are no current plans to limit the number of queries in flight. If JAVA-2561 doesn't meet your needs then please follow up with more information.

Ross

Comment by Ross Lawley [ 04/Sep/17 ]

Hi peterhendriks,

Apologies for the lack of response on this ticket, it appears to have fallen through the cracks.

I think providing a limit to the thread pool size or being able to provide your own ExecutorService should allow for finer grained control over memory resources. Do you agree? See: JAVA-2561

Ross

Generated at Thu Feb 08 08:55:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.