Summary

Add backoff to connection pool as described in the Technical Design: Support overload retryability in server-based clients (WRITING-28089) and the future work section of WRITING-32696.

Originally these changes were included in the scope of Avoid clearing the connection pool when the server connection rate limiter triggers (DRIVERS-3218) but we decided to push the pool backoff feature out into future work. During driver estimation we discovered pool backoff would add significant implementation costs to many drivers (Java, C, .NET) due to architectural design and various degrees of CMAP spec support. Because of this we decided to run the availability connection_storm_rate_limited_locust workload without pool backoff but with the "avoid pool clear" change. The results are identical meaning that the change in DRIVERS-3218 is sufficient enough to mitigate the connection storm/churn problem when the rate limiter is enabled. BaselineUser_Latency95thPercentile_degradation_percentage and BaselineUser_Latency95thPercentile_degradation_percentage are unchanged between the two (~3%). FTDC data also shows identical connection metrics.

Motivation

The motivation to introduce pool backoff is to add an additional layer of protection against connection storms. There may be edge cases where command retry backoff behavior and the connection rate limiter are insufficient and protecting from connection storms and pool backoff would help the system recover. When we pick this work up we should confirm this assumption via a new availability workload.

The other motivation is to ensure drivers and mongos behave the same. Mongos already implements the pool backoff behavior.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Acceptance Criteria

New availability workload demonstrating a clear benefit. Spec updates to CMAP.