Add backpressure labels to TCP/TLS connection establishment errors

ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Ruby Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      We add SystemOverloadedError and RetryableError to socket errors during the hello handshake, but that happens in PendingConnection#handshake!. Connection#create_socket does the TCP connect and TLS handshake before that, so errors raised there carry no labels and ConnectionPool#connect_connection marks the server Unknown and clears the pool.

      With TLS enabled, the ingress connection rate limiter rejects during the TLS handshake. That still clears the pool, which is the failure mode DRIVERS-3218 was meant to fix.

      CMAP says to label network errors and timeouts during connection establishment, including TLS I/O errors. DNS, certificate validation, hostname mismatch and SOCKS5 errors must stay unlabeled.

      The spec already has a prose test for this: "Connection Pool Backpressure" in server-discovery-and-monitoring-tests.md. It must run with and without TLS, and step 6 asserts zero PoolClearedEvent}}s. The Ruby test at {{spec/integration/sdam_prose_spec.rb only asserts the checkout failures and never checks for pool clears. The unified backpressure-network-* tests use a hello failpoint, so they do not cover the TLS path.

      Repro script below. It currently prints (Ruby 4.0.1, OpenSSL 3.6.0):

      tls reset:     Mongo::Error::SocketError cause=Errno::ECONNRESET labels=[]
tls eof:       Mongo::Error::SocketError cause=OpenSSL::SSL::SSLError labels=[]
certificate:   Mongo::Error::SocketError cause=OpenSSL::SSL::SSLError labels=[]
hostname:      Mongo::Error::SocketError cause=NilClass labels=[]
dns failure:   Mongo::Error::SocketError cause=Socket::ResolutionError labels=[]

      Expected: tls reset and tls eof carry both labels; the other three stay empty. TLS I/O errors can surface as Errno::* or OpenSSL::SSL::SSLError, so certificate errors cannot be excluded by type alone.

      Repro code:

      require 'mongo'
require 'openssl'
require 'socket'

def connect_error(uri)
  client = Mongo::Client.new(uri)
  server = client.cluster.instance_variable_get(:@servers).first
  connection = Mongo::Server::Connection.new(server, server.options.merge(generation: 0))
  connection.connect!
  nil
rescue StandardError => e
  e
ensure
  client&.close
end

def report(name, uri)
  error = connect_error(uri)
  labels = error.respond_to?(:labels) ? error.labels.sort : []
  puts format('%-14s %s cause=%s labels=%s',
              "#{name}:", error.class, error.cause.class, labels.inspect)
end

reset_server = TCPServer.new('127.0.0.1', 0)
reset_port = reset_server.addr[1]
Thread.new do
  loop do
    sock = reset_server.accept
    begin
      sock.readpartial(1024)
    rescue StandardError
      nil
    ensure
      sock.setsockopt(Socket::SOL_SOCKET, Socket::SO_LINGER, [1, 0].pack('ii'))
      sock.close
    end
  end
end

eof_server = TCPServer.new('127.0.0.1', 0)
eof_port = eof_server.addr[1]
Thread.new do
  loop do
    sock = eof_server.accept
    begin
      loop do
        break unless IO.select([sock], nil, nil, 0.05)

        sock.readpartial(4096)
      end
    rescue StandardError
      nil
    ensure
      sock.close
    end
  end
end

key = OpenSSL::PKey::RSA.new(2048)
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial = 1
name = OpenSSL::X509::Name.parse('/CN=localhost')
cert.subject = name
cert.issuer = name
cert.public_key = key.public_key
cert.not_before = Time.now - 60
cert.not_after = Time.now + 3600
cert.sign(key, OpenSSL::Digest::SHA256.new)

context = OpenSSL::SSL::SSLContext.new
context.cert = cert
context.key = key
cert_server = OpenSSL::SSL::SSLServer.new(TCPServer.new('127.0.0.1', 0), context)
cert_port = cert_server.to_io.addr[1]
Thread.new do
  loop do
    begin
      cert_server.accept.close
    rescue StandardError
      nil
    end
  end
end

report('tls reset', "mongodb://127.0.0.1:#{reset_port}/?tls=true&tlsAllowInvalidCertificates=true")
report('tls eof', "mongodb://127.0.0.1:#{eof_port}/?tls=true&tlsAllowInvalidCertificates=true")
report('certificate', "mongodb://localhost:#{cert_port}/?tls=true")
report('hostname', "mongodb://127.0.0.1:#{cert_port}/?tls=true&tlsAllowInvalidCertificates=true")
report('dns failure', 'mongodb://does-not-exist.invalid:27017/?tls=true')

puts "ruby #{RUBY_VERSION}, openssl #{OpenSSL::OPENSSL_VERSION}"

            Assignee:
            Unassigned
            Reporter:
            Dmitry Rybakov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: