Load-balanced mode permanently leaks a pooled connection when the initial find/aggregate command fails

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: 2.22.0
    • Component/s: Connections, Load Balancer
    • None
    • None
    • Ruby Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Related: PR https://github.com/mongodb/mongo-ruby-driver/pull/3041 ("Don't leak connections in load-balanced topology", merged 2026-05-20, shipped in 2.24.1) fixed the same class of missing-check-in leak on the cleanup path – the cursor reaper skipping check_in when killCursors failed over a dead pinned connection – and added the rescue to change_stream.rb. The leak reported here is distinct: it occurs before any cursor exists, so neither the finalizer, the reaper, nor 3041's fix can reach it. This report completes the remaining sites of the pattern 3041 started fixing. See also RUBY-3463 for the pinning design this bug lives inside.

       

      In load-balanced mode, send_initial_query checks a connection out of the pool before executing the initial query operation:

        # lib/mongo/collection/view/iterable.rb (v2.22.0)
        def send_initial_query(server, context)
          operation = initial_query_op(context.session)
          if server.load_balancer?
            # Connection will be checked in when cursor is drained.
            connection = server.pool.check_out(context: context)
            operation.execute_with_connection(connection, context: context)
          else
            operation.execute(server, context: context)
          end
        end 

      The comment's assumption — the connection is checked in when the cursor is drained — only holds if the command succeeds. If execute_with_connection raises (server-side   OperationFailure, socket timeout, MaxTimeMSExpired, etc.), no Cursor object exists yet, so nothing owns the connection: there is no rescue/ensure that checks it back in,   Cursor#close/check_in_connection are unreachable, and the cursor reaper never sees it (a KillSpec carrying the connection is only registered by the cursor finalizer, which requires a cursor to have been created). The pool slot is lost permanently for the life of the process.

      The same pattern exists at every load-balanced initial-command site, verified on current master (8222a3d, 2026-08-19):

      collection/view/iterable.rb (find),

      collection/view/aggregation.rb (aggregate),

      collection/view/map_reduce.rb (mapReduce),

      collection/view/readable.rb (parallel scan getMore),

      database/view.rb (listCollections),

      index/view.rb (listIndexes)

      The driver already handles this failure mode correctly in ChangeStream and Database#cursor_command; this report covers the six sites lacking the same protection.

       

      Impact: In load-balanced topology, every failed initial command permanently shrinks the effective pool. Under production load with intermittent server-side errors, processes progressively exhaust their pool and end in a terminal state where every operation raises Mongo::Error::ConnectionCheckOutTimeout (0 available, N checked out, 0 pending (max size: N)), requiring a process restart. We traced weeks of recurring incidents to this.

      Reproduction (against any loadBalanced=true deployment):

      client = Mongo::Client.new(uri_with_load_balanced_true, min_pool_size: 5, max_pool_size: 30)
      pool = client.cluster.servers.first.pool
      stat = ->(l) { puts "#{l}: size=#{pool.size} available=#{pool.available_count}" }  client[:anything].find(_id: BSON::ObjectId.new).to_a  # warm up
      stat.call('baseline')  3.times { client[:anything].aggregate([{ '$bogus' => 1 }]).to_a rescue nil }
      stat.call('after 3 failed initial commands')
      GC.start; GC.start; sleep 12 # give finalizers + cursor reaper time
      stat.call('after GC + reaper')
      # Observed (deterministic across runs):  baseline:                       size=5 available=5
      after 3 failed initial commands: size=5 available=2
      after GC + reaper:               size=5 available=2 # slots never recovered 

      Expected: available returns to 5 — a failed initial command should check the connection back in before propagating the error.

      For contrast, an abandoned cursor (opened, partially iterated, dropped without close) is correctly recovered after GC via the finalizer → CursorReaperconnection_pool.check_in path. The gap is exclusively the window between pool.check_out and cursor construction.

      Suggested fix: apply the pattern the driver already uses in ChangeStream and Database#cursor_command to all six sites:

      connection ||= server.pool.check_out(context: context)
      begin
        operation.execute_with_connection(connection, context: context)
      rescue StandardError
        server.pool.check_in(connection) unless connection.pinned?
        raise
      end

      ensure is not appropriate — on success the connection must remain checked out for the cursor. Transaction-pinned connections must stay checked out even on failure, matching Database#cursor_command.

            Assignee:
            Jamis Buck
            Reporter:
            Dmytro Rymar (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: