Handshake network errors are always labeled SystemOverloadedError, so the server is never marked Unknown from connection-establishment failures

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Ruby Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Observed behavior

      Server::PendingConnection#handshake! (lib/mongo/server/pending_connection.rb:172-175) adds the SystemOverloadedError and RetryableError labels to every Mongo::Error::SocketError / Mongo::Error::SocketTimeoutError raised during the initial hello handshake, unconditionally:

      if e.is_a?(::Mongo::Error) && (e.is_a?(Error::SocketError) || e.is_a?(Error::SocketTimeoutError))
        e.add_label('SystemOverloadedError')
        e.add_label('RetryableError')
      end
      

      Both SDAM error paths that would mark the server Unknown gate on the absence of that label:

      • Server#handle_handshake_failure! (lib/mongo/server.rb) calls unknown! only unless e.label?('SystemOverloadedError').
      • ConnectionPool#connect_connection (lib/mongo/server/connection_pool.rb) calls unknown! only if e.is_a?(Mongo::Error) && !e.label?('SystemOverloadedError').

      Because the label is always present on handshake network errors, neither path ever marks the server Unknown for a genuine network failure during connection establishment. The server is only marked Unknown later, when the monitor's own check fails.

      Why this looks wrong

      The SystemOverloadedError label is a server-provided signal, per the Client Backpressure spec (client-backpressure.md, "SystemOverloadedError label"): it is attached by the server to overload responses (e.g. IngressRequestRateLimitExceeded, code 462). It is not something the driver should synthesize for arbitrary network errors.

      • The ingress request rate limiter only applies to authenticated connections, so a pre-auth handshake will not hit it (client-backpressure.md, "Why not apply the overload policy to authentication commands...").
      • Connection-establishment commands are explicitly excluded from the overload retry policy (client-backpressure.md, "Driver mechanisms subject to the retry policy").
      • SDAM requires marking the server Unknown on a network error/timeout during the handshake ("MongoDB Handshake errors", server-discovery-and-monitoring.md).

      Synthesizing SystemOverloadedError on all handshake socket errors therefore both (a) mislabels ordinary network failures as server-overload backpressure, and (b) suppresses the spec-required "mark Unknown on handshake network error" behavior.

      Origin and related work

      The labeling was introduced by RUBY-3700 ("Do not clear pool on retryable errors", commit cc3722956). The gray area is distinguishing the server's connection rate limiter (which can trigger during establishment and should not mark the server Unknown) from ordinary network failures. The spec notes the connection-rate-limiter case but says drivers "already behave in an appropriate manner" for monitoring/RTT and does not prescribe synthesizing the label for application connections.

      This surfaced during RUBY-3909 review: the network_error: true branch added there for the handshake path is effectively unreachable today because of this labeling.

      Proposed investigation

      • Confirm against the Client Backpressure spec and a reference driver (pymongo) how the ingress connection rate limiter is meant to be detected during establishment, and whether the driver should ever synthesize SystemOverloadedError locally.
      • If synthesizing is incorrect, scope the label to actual server-signalled overload responses so genuine network failures during the handshake once again mark the server Unknown and clear the pool per SDAM.
      • Add a regression test asserting that a network error during the initial hello marks the server Unknown.

      Severity is a first estimate; please confirm during triage.

            Assignee:
            Dmitry Rybakov
            Reporter:
            Dmitry Rybakov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: