[JAVA-3927] Rate limit new connection creations (maxConnecting) Created: 10/Jan/21  Updated: 28/Oct/23  Resolved: 14/Apr/21

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: None
Fix Version/s: 4.3.0

Type: New Feature Priority: Major - P3
Reporter: Esha Bhargava Assignee: Valentin Kavalenka
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
causes JAVA-4316 The background thread should be able ... Closed
causes JAVA-4451 Fix the await condition in OpenConcur... Closed
causes JAVA-4104 Fix AbstractConnectionPoolTest failures Closed
causes JAVA-4146 Fix timeout handling in `DefaultConne... Closed
causes JAVA-4452 Improve performance characteristics o... Closed
causes JAVA-4318 Reduce the probability of failing the... Closed
Related
is related to JAVA-4390 Make maxConnecting configurable Closed
Epic Link: Avoiding connection storms
Quarter: FY22Q4
Backwards Compatibility: Fully Compatible
Documentation Changes: Not Needed

 Comments   
Comment by senthil kumar c [ 23/Jan/23 ]

Hi Ashni,

Thanks for the reply. I have created new ticket as below,
https://jira.mongodb.org/browse/JAVA-4851

Thanks,

Senthil

Comment by Ashni Mehta [ 17/Jan/23 ]

Hi Senthil, I chatted with the team about this one. Regarding the commit that you linked about deadlock, we don't believe that it is applicable specifically to the 3.12.9 driver version.

Regarding the similar issue that you mention, I think it would be best to file a new ticket on this project with your driver/server version details and any additional information that you think might be helpful for us as we investigate.

Comment by senthil kumar c [ 13/Jan/23 ]

Hi Ashni,

It would be helpful us, if you provide an update on the above query.

Thanks,

Senthil

Comment by senthil kumar c [ 06/Jan/23 ]

Hi Ashni,

Thanks for the quick response. My query is, whether the reported issue is applicable for 3.12.9?
Because, we are using 3.12.9 in our product, and we are seeing similar issue. So, if this is applicable, do we need to backport the fix in 3.12.9?

Thanks,

Senthil

Comment by Ashni Mehta [ 05/Jan/23 ]

Hi Senthil, I believe this is available in 4.3.0 and later.

Comment by senthil kumar c [ 05/Jan/23 ]

Hi All,

Issue fixed as part of the below is applicable for 3.12.9 version?
https://github.com/mongodb/mongo-java-driver/commit/73e30857eed2f1d02ff7e9a6e5028d557a1cf4cf

Can someone help on this.

Comment by Githook User [ 08/Jun/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Refactor `DefaultConnectionPool` to use single worker thread when checking out asynchronously (#722)

When implementing https://github.com/mongodb/mongo-java-driver/commit/73e30857eed2f1d02ff7e9a6e5028d557a1cf4cf
I correctly explained the deadlock situation:
1) An async task that created a non-open connection and tries to open it (let us call it an "open task")
may block another async task that tries to create a connection (let us call it a "get task")
if `maxSize` is reached.
2) If both tasks are submitted to the same queue such that the get task is ahead of the open task, then
such a get task naturally blocks the open task while being blocked by the open task.
but failed to optimally fix it.

The solution to the problem I used is to submit tasks to different queues, which,
because both tasks are blocking, required introducing one more async worker thread.

However, there is another approach to solving the deadlock problem,
which did not occur to me at that time: prevent the get task to be ahead of the open task
in the queue. This may happen due to the optimization in the `getAsync` method, which
tries to check out synchronously if an open connection is available,
i.e., if checking it out requires no blocking. It is a nice optimization that I
tried to preserve (it was there before my changes) when implemented JAVA-3927.
However, this optimization opens up opportunities for a get task being blocked
by an open task to be ordered ahead of the open task in the queue.

If I realized that previously, I would have sacrificed this optimization
to eliminate the deadlock instead of introducing another queue and a thread.
Fortunately, the it occurred to me now.

This change also fixes reactive `MainTransactionsTest` runner
so that the `abort.json`: "implicit abort" test succeeds.
This test expects command_started_event for the abortTransaction command, which is sent
as a result of closing the session used by the test. The close method is asynchronous and
the test must wait for its completion.

JAVA-4146
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/6ab9ce6462f5cc1a994e9a829a1a7c27ecb2c546

Comment by Githook User [ 02/May/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Simplify `DefaultConnectionPool.OpenConcurrencyLimiter.tryLock` (#702)

When locking for the purpose of using the wait/signal approach,
one does not have to account the time spent acquiring the lock
towards the timeout. While doing so may reduce the
non-deterministic duration by which a timeout is exceeded
by any implementation of a timeout, such overtime is usually negligible,
unless the lock is highly contended and/or checking the condition
before starting to wait is time-consuming.

The Java SE example for `java.util.concurrent.locks.Condition.awaitNanos`
ignores the time spent acquiring the lock. Doing so potentially reduces
responsiveness to timeouts, but does not
affect other performance characteristics, so I believe such simplification
is worth it and I should have taken this path initially.

JAVA-3927
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/ddefdb0cc29541f82d507318ef7f94ec03efcd39

Comment by Githook User [ 01/May/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Fix deadlock and couple more problems in `DefaultConnectionPool` (#699)

This commit fixes three problems in `DefaultConnectionPool`/`ConcurrentPool`:

  • deadlock in `DefaultConnectionPool`;
  • unnecessary and harmful removal from `OpenConcurrencyLimiter.desiredConnectionSlots`;
  • double release in `DefaultConnectionPool`/`ConcurrentPool`.

Deadlock in `DefaultConnectionPool`

a) Before introducing OpenConcurrencyLimiter, "AsyncGetter" thread was used only
to do blocking `ConcurrentPool.get`.
b) After, I started to additionally use "AsyncGetter" to do
blocking `OpenConcurrencyLimiter.waitUntilOpenPermitAvailable`.

As a result, we may have a thread that gets the last connection (`maxSize` is reached)
from `ConcurrentPool` and submits `waitUntilOpenPermitAvailable` to "AsyncGetter".
Concurrently with this happening, the "AsyncGetter" tries to get from `ConcurrentPool`
and is blocked because there are no more connections available. In such an execution,
`waitUntilOpenPermitAvailable` cannot be completed by "AsyncGetter" because
"AsyncGetter" is blocked doing a different task, which itself cannot be completed.

A solution is to do `ConcurrentPool.get` and `waitUntilOpenPermitAvailable`
in different threads. This way these two different kinds of tasks
will not block each other by waiting in the same queue to be done by a single thread.

Potential unnecessary and harmful removal from `OpenConcurrencyLimiter.desiredConnectionSlots`

If `acquirePermitOrGetAvailableOpenedConnection` is called with `true` as `tryGetAvailable`,
and `getPooledConnectionImmediately` throws an exception,
then `expressDesireToGetAvailableConnection` is not called
but `giveUpOnTryingToGetAvailableConnection` is still called, which is incorrect.

Double release in `DefaultConnectionPool`/`ConcurrentPool`

On one hand `ConcurrentPool.ensureMinSize` tries to not require the caller to do
what can be done by the method itself, e.g., it releases the connection to the pool itself.
On the other hand, always releasing in `ensureMinSize` may lead
to releasing a permit for the same connection twice, thus not respecting the `maxSize`.

It is not easy (requires an additional knob and logic)
to prevent the caller (`DefaultConnectionPool`) from releasing a connection
when initialization fails. It is, therefore, seems better to mandate that the caller
releases a connection if initialization fails.

JAVA-3927
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/73e30857eed2f1d02ff7e9a6e5028d557a1cf4cf

Comment by Githook User [ 14/Apr/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Rate limit new connection creations (maxConnecting) (#685)

JAVA-3927
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/ead0357131c1c0baa364656b07c91f2c789918e3

Generated at Thu Feb 08 09:00:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.