ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Dotnet Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      *Driver:* 3.10.0 (same code in 3.5.0-3.13.0 and current `main`); also 2.24.0 for the pool part
      *Runtime:* .NET 10 on Linux, `net6.0+` build of the driver, `tls=true`, replica set
      *Server:* MongoDB 7.0 with `--tlsMode requireTLS`
       

          1. Summary
            When a node accepts TCP connections but never answers (for example, its old connections are dropped silently without a RST), the TLS handshake of a new connection waits forever. This affects two code paths:

      1. *Server monitor (regression in 3.5.0).* The monitor reconnects and blocks in the synchronous `SslStream.AuthenticateAsClient`. The node is never marked as available again, and every operation fails with `A timeout occurred after 30000ms selecting a server`. On Linux the only thing that ends it is the kernel's default TCP keepalive, about 2h11m. If the peer keeps answering at TCP level, nothing ends it.
      2. *Pool connection opens (all versions).* An operation that opens a connection during the hang waits in `SslStream.AuthenticateAsClientAsync`, which is bounded only by the operation's cancellation token. With `CancellationToken.None`, or a token that is not cancelled, it never ends. Two such opens take both `maxConnecting` slots. After the node is healthy again, every later operation that needs a connection fails with `Timed out waiting in connecting queue` until the process restarts. On 2.24.0 the slot is not released even when the token is cancelled.

      We hit this in production: after a replica set node became unresponsive and its connections were dropped silently, long-running services could not select a server for about 2h20m, until they were restarted. Short-lived processes recovered on their own.

          1. Stuck monitor thread (3.10.0, `dotnet-stack`)

      ```
      System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions)
      MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint, CancellationToken)
      MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(OperationContext)
      MongoDB.Driver.Core.Connections.BinaryConnection.Open(OperationContext)
      MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(OperationContext)
      MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken)
      MongoDB.Driver.Core.Servers.ServerMonitor.MonitorServer(CancellationToken)
      ```
       

          1. Cause
      • `connectTimeoutMS` bounds only the TCP connect (`TcpStreamFactory`). Reads after the stream is created are bounded in `BinaryConnection` (`socketTimeoutMS`, operation timeout). The TLS handshake runs between the two and has no bound of its own.
      • *Monitor:* up to 3.4.x, `TcpStreamFactory` set `NetworkStream.ReadTimeout`, and the monitor stream used the heartbeat socket timeout (default `connectTimeout`, 30s). That bounded the synchronous handshake. In 3.5.0, 1720(https://github.com/mongodb/mongo-csharp-driver/pull/1720) (CSHARP-3538) set the monitor stream to `readTimeout: null`, and 1758(https://github.com/mongodb/mongo-csharp-driver/pull/1758) (CSHARP-5695) removed `ReadTimeout`/`WriteTimeout` from `TcpStreamFactory`. `AuthenticateAsClient` takes no token, so the `connectTimeout` in the monitor's `OperationContext` does not reach it. `serverMonitoringMode=poll` uses the same path.
      • *Pool:* `BinaryConnection.OpenHelperAsync` passes only `operationContext.CombinedCancellationToken` to `SslStreamFactory.CreateStreamAsync`. Without a cancelled token or an operation timeout, the handshake has no limit. `timeoutMS` cannot be used as a workaround in released builds: it is `internal`, and `ConnectionString` parses it only under `#if DEBUG`.
          1. Reproduction

      1. Run a TLS `mongod` and put a TCP proxy in front of it. On command, the proxy stops forwarding: existing connections stay open but no bytes pass, and new connections are accepted but never answered.
      2. Connect a client through the proxy with `tls=true` and default settings. Every 10s, run `ping`, once with `CancellationToken.None` and once with a 5s token.
      3. Switch the proxy to hang for 150s, then forward new connections again while the old ones stay frozen (or drop them silently).

      Client Result after the node is reachable again (observed for 4-5 minutes)
      — —
      3.10.0, defaults Stuck, with and without a token: monitor blocked in `AuthenticateAsClient`, server selection timeouts
      2.24.0, defaults Monitor recovers in ~10s, but operations stay stuck, with and without a token: 2 pool opens never finish, `Timed out waiting in connecting queue`
      3.10.0 + `Socket.ReceiveTimeout` 60s (workaround below), `CancellationToken.None` Monitor recovers, operations stay stuck on the 2 pool opens
      3.10.0 + `Socket.ReceiveTimeout` 60s, 5s token Recover in 3-5s

      The difference between the two handshake paths is .NET behaviour: `Socket.ReceiveTimeout` applies only to synchronous reads. Against a server that accepts TCP and never sends:

      ```
      sync AuthenticateAsClient, Socket.ReceiveTimeout 2s -> IOException after 2.0s
      async AuthenticateAsClientAsync, Socket.ReceiveTimeout 2s -> still waiting after 15.0s
      async AuthenticateAsClientAsync, token cancelled after 2s -> OperationCanceledException after 2.0s
      ```

          1. Workaround we use
      • `ClusterConfigurator` → `ConfigureTcp(socketConfigurator: ...)` with `Socket.ReceiveTimeout = 60s`, which bounds the synchronous monitor handshake, plus TCP keepalive 30s/10s/3, which closes connections to a peer that disappeared.
      • A deadline token on every operation, so stuck pool opens end.
          1. Expected

      Every connection establishment (TCP connect, TLS handshake and the initial `hello`/auth), for both monitor and pool connections, should be bounded by `connectTimeoutMS` regardless of the caller's token. For example, run the monitor handshake asynchronously with a token derived from the `OperationContext` timeout, and link `connectTimeout` into the token passed to `CreateStreamAsync` for pool connections. The Go driver does this: it wraps the whole connect in a `connectTimeout` context and uses `tls.Conn.HandshakeContext`.

      Possibly related: CSHARP-1888, CSHARP-1105, JAVA-6302.

            Assignee:
            Oleksandr Poliakov
            Reporter:
            aleksandr.gribanov@idt.net
            None
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: