-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
None
-
Dotnet Drivers
-
None
-
None
-
None
-
None
-
None
-
None
*Driver:* 3.10.0 (same code in 3.5.0-3.13.0 and current `main`); also 2.24.0 for the pool part
*Runtime:* .NET 10 on Linux, `net6.0+` build of the driver, `tls=true`, replica set
*Server:* MongoDB 7.0 with `--tlsMode requireTLS`
-
-
- Summary
When a node accepts TCP connections but never answers (for example, its old connections are dropped silently without a RST), the TLS handshake of a new connection waits forever. This affects two code paths:
- Summary
-
1. *Server monitor (regression in 3.5.0).* The monitor reconnects and blocks in the synchronous `SslStream.AuthenticateAsClient`. The node is never marked as available again, and every operation fails with `A timeout occurred after 30000ms selecting a server`. On Linux the only thing that ends it is the kernel's default TCP keepalive, about 2h11m. If the peer keeps answering at TCP level, nothing ends it.
2. *Pool connection opens (all versions).* An operation that opens a connection during the hang waits in `SslStream.AuthenticateAsClientAsync`, which is bounded only by the operation's cancellation token. With `CancellationToken.None`, or a token that is not cancelled, it never ends. Two such opens take both `maxConnecting` slots. After the node is healthy again, every later operation that needs a connection fails with `Timed out waiting in connecting queue` until the process restarts. On 2.24.0 the slot is not released even when the token is cancelled.
We hit this in production: after a replica set node became unresponsive and its connections were dropped silently, long-running services could not select a server for about 2h20m, until they were restarted. Short-lived processes recovered on their own.
-
-
- Stuck monitor thread (3.10.0, `dotnet-stack`)
-
```
System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions)
MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint, CancellationToken)
MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(OperationContext)
MongoDB.Driver.Core.Connections.BinaryConnection.Open(OperationContext)
MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(OperationContext)
MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken)
MongoDB.Driver.Core.Servers.ServerMonitor.MonitorServer(CancellationToken)
```
-
-
- Cause
-
- `connectTimeoutMS` bounds only the TCP connect (`TcpStreamFactory`). Reads after the stream is created are bounded in `BinaryConnection` (`socketTimeoutMS`, operation timeout). The TLS handshake runs between the two and has no bound of its own.
- *Monitor:* up to 3.4.x, `TcpStreamFactory` set `NetworkStream.ReadTimeout`, and the monitor stream used the heartbeat socket timeout (default `connectTimeout`, 30s). That bounded the synchronous handshake. In 3.5.0, 1720(https://github.com/mongodb/mongo-csharp-driver/pull/1720) (
CSHARP-3538) set the monitor stream to `readTimeout: null`, and 1758(https://github.com/mongodb/mongo-csharp-driver/pull/1758) (CSHARP-5695) removed `ReadTimeout`/`WriteTimeout` from `TcpStreamFactory`. `AuthenticateAsClient` takes no token, so the `connectTimeout` in the monitor's `OperationContext` does not reach it. `serverMonitoringMode=poll` uses the same path. - *Pool:* `BinaryConnection.OpenHelperAsync` passes only `operationContext.CombinedCancellationToken` to `SslStreamFactory.CreateStreamAsync`. Without a cancelled token or an operation timeout, the handshake has no limit. `timeoutMS` cannot be used as a workaround in released builds: it is `internal`, and `ConnectionString` parses it only under `#if DEBUG`.
-
- Reproduction
-
1. Run a TLS `mongod` and put a TCP proxy in front of it. On command, the proxy stops forwarding: existing connections stay open but no bytes pass, and new connections are accepted but never answered.
2. Connect a client through the proxy with `tls=true` and default settings. Every 10s, run `ping`, once with `CancellationToken.None` and once with a 5s token.
3. Switch the proxy to hang for 150s, then forward new connections again while the old ones stay frozen (or drop them silently).
| Client | Result after the node is reachable again (observed for 4-5 minutes) |
| — | — |
| 3.10.0, defaults | Stuck, with and without a token: monitor blocked in `AuthenticateAsClient`, server selection timeouts |
| 2.24.0, defaults | Monitor recovers in ~10s, but operations stay stuck, with and without a token: 2 pool opens never finish, `Timed out waiting in connecting queue` |
| 3.10.0 + `Socket.ReceiveTimeout` 60s (workaround below), `CancellationToken.None` | Monitor recovers, operations stay stuck on the 2 pool opens |
| 3.10.0 + `Socket.ReceiveTimeout` 60s, 5s token | Recover in 3-5s |
The difference between the two handshake paths is .NET behaviour: `Socket.ReceiveTimeout` applies only to synchronous reads. Against a server that accepts TCP and never sends:
```
sync AuthenticateAsClient, Socket.ReceiveTimeout 2s -> IOException after 2.0s
async AuthenticateAsClientAsync, Socket.ReceiveTimeout 2s -> still waiting after 15.0s
async AuthenticateAsClientAsync, token cancelled after 2s -> OperationCanceledException after 2.0s
```
-
-
- Workaround we use
-
- `ClusterConfigurator` → `ConfigureTcp(socketConfigurator: ...)` with `Socket.ReceiveTimeout = 60s`, which bounds the synchronous monitor handshake, plus TCP keepalive 30s/10s/3, which closes connections to a peer that disappeared.
- A deadline token on every operation, so stuck pool opens end.
-
-
- Expected
-
Every connection establishment (TCP connect, TLS handshake and the initial `hello`/auth), for both monitor and pool connections, should be bounded by `connectTimeoutMS` regardless of the caller's token. For example, run the monitor handshake asynchronously with a token derived from the `OperationContext` timeout, and link `connectTimeout` into the token passed to `CreateStreamAsync` for pool connections. The Go driver does this: it wraps the whole connect in a `connectTimeout` context and uses `tls.Conn.HandshakeContext`.
Possibly related: CSHARP-1888, CSHARP-1105, JAVA-6302.
- is related to
-
CSHARP-6305 TLS handshake has no timeout: a frozen server node hangs new connections until TCP keepalive (~2 h on Linux)
-
- Investigating
-