Section "Checking Out a Connection" in the CMAP specification says:
the pool MUST wait to service the request until ... or until a Connection becomes available
the Pool MUST NOT service any newer checkOut requests before fulfilling the original one which could not be fulfilled
Currently, DefaultConnectionPool.OpenConcurrencyLimiter.acquirePermitOrGetAvailableOpenedConnection uses a hand over mechanism to implement the above requirements: when a connection is checked in, threads waiting inside acquirePermitOrGetAvailableOpenedConnection are signalled, and are allowed to grab checked in connections before any other thread has a chance to check them out. However, there is another mechanism that may result in available connections appearing in a pool: the background thread populating the pool based on the minSize. Consider the following execution:
- tB is the background thread that is establishing a connection to satisfy minSize.
- t1 is a thread that is establishing a connection to satisfy a checkout operation.
- t2 is a thread that is waiting for a permit to establish a connection to satisfy a checkout, because both permits are taken by tB and t1.
- tB completes establishing a connection, t1 continues establishing.
- t2 can now either notice that there is a new connection available in the pool and check it out, or take the available permit and start establishing a new connection.
According to the specification, t2 MUST be the one who gets the connection created by tB, but currently it ignores that connection completely, because the background thread populating the pool does not check new connections in. Instead, populating uses the ConcurrentPool.ensureMinSize method.
I believe, the background thread (see DefaultConnectionPool.doMaintenance) should be able to use DefaultConnectionPool.OpenConcurrencyLimiter.tryHandOverOrRelease (with some modifications) in order to hand over new connections similarly to how this happens in DefaultConnectionPool.PooledConnection.close. If the background thread successfully hands over a connection, then ConcurrentPool.ensureMinSize must not call ConcurrentPool.release for such a connection.
This scenario and the way to fix it came to me while I was thinking about
JAVA-4288. Fixing this bug will not only make the driver more compliant with the specification, but also will help reducing checkout latencies in situations like that in JAVA-4288. However, on its own, it is unlikely to resolve JAVA-4288.