Summary

Depending on the implementation details of the connection pool WaitQueue, it's possible for the sequence of operations described in pool-checkout-returned-connection-maxConnecting.yml to result in either 3 or 4 created connections.

The sequence of events expected by the test appears to be:

1. Check out a connection, which triggers the 1st connection creation, and hold onto it.
2. Start 3 threads and check out a connection in each thread.
3. All 3 check-out requests enter the wait queue and the first 2 (bounded by maxConnecting) start to create new connections.
4. Check in the connection held onto in step #1.
5. The checked-in connection is delivered to the check-out request not waiting on a new connection (see comment here).
6. The 2 remaining check-out requests are satisfied by the 2nd and 3rd new connections.
7. 3 connections are created for 4 check-out requests.

Here's a timing diagram of that sequence. The "N" indicates a check-out causing a new connection to start establishment. The numbers are the connection IDs and indicate which new connection is delivered to which waiting check-out. Note that connection 1 is the connection created by the main thread in step #1 above.

check-out 1 |-N-------------2
new conn 2  | 2-------------2
            |
check-out 2 |--N--------------3
new conn 3  |  3--------------3
            |
check-out 3 |--[check in]1

However, the CMAP spec description of WaitQueue delegates specific implementation to the drivers as long as it satisfies the described behaviors. Specifically, the driver must deliver available connections to waiting check-out requests in the order that they came in:

When Connections are made available, they are issued out to threads in the order that the threads entered the WaitQueue.

This is where the Go driver behavior deviates from most other drivers. In the Go driver, all connections are established in background threads, so a check-out request can cause a new connection to start establishment, then have a connection delivered to it by a check-in. The connection establishment continues in the background, and the new connection is delivered to the first waiting check-out when it's ready. That can create the following sequence which is also valid, but fails the test:

1. Check out a connection, which triggers the 1st connection creation, and hold onto it.
2. Start 3 threads and check out a connection in each thread.
3. All 3 check-out requests enter the wait queue and the first 2 (bounded by maxConnecting) start to create new connections.
4. Check in the connection held onto in step #1.
5. The checked-in connection is delivered to the first check-out request (the one waiting on the 2nd new connection).
6. The 2nd new connection becomes available, but the original check-out request is no longer waiting (the checked-in connection was delivered to it), so it's delivered to the check-out request waiting on the 3rd new connection.
7. The number of in-progress connections is now below maxConnecting and there is a check-out request waiting, so start creating a 4th new connection.
8. The 3rd new connection becomes available, but the original check-out request is no longer waiting (the 2nd new connection was delivered to it), so it's delivered to the check-out request waiting for the 4th new connection.
9. The 4th new connection becomes available, but there are no awaiting check-out requests, so check it into the pool as available.
10. 4 connections are created for 4 check-out requests.

Here's a timing diagram of that sequence. The "N" indicates a check-out causing a new connection to start establishment. The numbers are the connection IDs and indicate which new connection is delivered to which waiting check-out. Note that connection 1 is the connection created by the main thread in step #1 above.

check-out 1 |-N-[check in]1
new conn 2  | 2-------------2
            |               2
check-out 2 |--N------------2
new conn 3  |  3--------------3
            |                 3
check-out 3 |---------------N-3
new conn 4  |               4--------------4

Motivation

Who is the affected end user?

Drivers devs implementing maxConnecting and trying to run CMAP spec tests.

How does this affect the end user?

There is no known end user impact.

How likely is it that this problem or use case will occur?

The 2nd described behavior currently happens reliably in the Go driver. The impact to other drivers is unknown, but it seems like most other drivers' connection pool implementations work with the existing test. The impact may increase as other drivers make connection pool implementation changes as part of CSOT.

If the problem does occur, what are the consequences and how severe are they?

The CMAP spec tests may fail. There is no known end user impact.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

Yes.