[JAVA-4316] The background thread should be able to hand over a new connection to a thread that is blocked by the max connecting limit Created: 27/Sep/21  Updated: 28/Oct/23  Resolved: 15/Oct/21

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: 4.3.0
Fix Version/s: 4.3.4

Type: Bug Priority: Major - P3
Reporter: Valentin Kavalenka Assignee: Valentin Kavalenka
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt Dependency
has to be done before JAVA-4346 Add a CMAP test that verifies the bac... Closed
Problem/Incident
causes DRIVERS-1948 Add a CMAP test that verifies the bac... Implementing
is caused by JAVA-3927 Rate limit new connection creations (... Closed
Related
is related to JAVA-4288 Allow configuration of MAX_CONNECTING... Closed
Epic Link: Avoiding connection storms
Quarter: FY22Q4
Backwards Compatibility: Fully Compatible
Documentation Changes: Not Needed

 Description   

Section "Checking Out a Connection" in the CMAP specification says:

the pool MUST wait to service the request until ... or until a Connection becomes available

the Pool MUST NOT service any newer checkOut requests before fulfilling the original one which could not be fulfilled

Currently, DefaultConnectionPool.OpenConcurrencyLimiter.acquirePermitOrGetAvailableOpenedConnection uses a hand over mechanism to implement the above requirements: when a connection is checked in, threads waiting inside acquirePermitOrGetAvailableOpenedConnection are signalled, and are allowed to grab checked in connections before any other thread has a chance to check them out. However, there is another mechanism that may result in available connections appearing in a pool: the background thread populating the pool based on the minSize. Consider the following execution:

  1. tB is the background thread that is establishing a connection to satisfy minSize.
  2. t1 is a thread that is establishing a connection to satisfy a checkout operation.
  3. t2 is a thread that is waiting for a permit to establish a connection to satisfy a checkout, because both permits are taken by tB and t1.
  4. tB completes establishing a connection, t1 continues establishing.
  5. t2 can now either notice that there is a new connection available in the pool and check it out, or take the available permit and start establishing a new connection.

According to the specification, t2 MUST be the one who gets the connection created by tB, but currently it ignores that connection completely, because the background thread populating the pool does not check new connections in. Instead, populating uses the ConcurrentPool.ensureMinSize method.

I believe, the background thread (see DefaultConnectionPool.doMaintenance) should be able to use DefaultConnectionPool.OpenConcurrencyLimiter.tryHandOverOrRelease (with some modifications) in order to hand over new connections similarly to how this happens in DefaultConnectionPool.PooledConnection.close. If the background thread successfully hands over a connection, then ConcurrentPool.ensureMinSize must not call ConcurrentPool.release for such a connection.

This scenario and the way to fix it came to me while I was thinking about JAVA-4288. Fixing this bug will not only make the driver more compliant with the specification, but also will help reducing checkout latencies in situations like that in JAVA-4288. However, on its own, it is unlikely to resolve JAVA-4288.



 Comments   
Comment by Githook User [ 15/Oct/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Enable the background thread to hand over a newly opened connection to a thread that is blocked by the max connecting limit (#805)

Backport JAVA-4316, JAVA-4346
Branch: 4.3.x
https://github.com/mongodb/mongo-java-driver/commit/1fb7ed33b96f99b222a675b20ee53148b08a9cad

Comment by Githook User [ 15/Oct/21 ]

Author:

{'name': 'Valentin Kovalenko', 'email': 'valentin.kovalenko@mongodb.com', 'username': 'stIncMale'}

Message: Enable the background thread to hand over a newly opened connection to a thread that is blocked by the max connecting limit (#799)

JAVA-4316, JAVA-4346
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/1459dba02430d92d837d1d605d672379a76afaac

Generated at Thu Feb 08 09:01:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.