Uploaded image for project: 'Ruby Driver'
  1. Ruby Driver
  2. RUBY-3284

Connection Pool does not open new connections when needed

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 2.19.1
    • Affects Version/s: 2.19.0
    • Component/s: Connections
    • Labels:
      None

      Hi,

      I just upgraded from 2.18.2 to 2.19.0 3 days ago, and very quickly noticed a surge of Mongo::Error::ConnectionCheckOutTimeout errors. I let it run for a couple days as I wasn't sure what was the cause yet and suspected poor server performance, but today I tried to revert the mongo bump (at 14:47) and the errors immediately stopped (last one at 14:43), confirming IMO that some change in 2.19.0 is causing it. I had 2345 occurences in total across the 3 days that 2.19.0 have been in production.

      Here is the exception in more details:

      I'm running Sidekiq with 80 threads, so I configured my Mongoid pool to 100:

      Unable to find source-code formatter for language: yml. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      ---
      :concurrency: 80
      :queues:
        - mailers
        - default
        - low
      
      Unable to find source-code formatter for language: yml. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      production:
        clients:
          default:
            database: updown
            hosts:
              - db1.updn.io
              - db2.updn.io
              - db3.updn.io
            options:
              user: updown
              password: <%= Rails.application.credentials.dig(:mongodb, :production) %>
              compressors: [zstd, zlib]
              retry_writes: true
              max_read_retries: 3
              read:
                mode: :primary_preferred
              max_pool_size: 100
              wait_queue_timeout: 5
              socket_timeout: 60
      

      Under these condition there is no reason to hit this exception ever, and as a matter of fast I never did for more than 2 years!

      I tried to look at the code diff but it's enormous so there's way I'll be able to find the reason on my own. Do you have any idea what code change in 2.19.0 could be causing this?

        1. image-2023-06-30-21-11-35-220.png
          image-2023-06-30-21-11-35-220.png
          374 kB
        2. screenshot-1.png
          screenshot-1.png
          23 kB
        3. screenshot-2.png
          screenshot-2.png
          73 kB

            Assignee:
            dmitry.rybakov@mongodb.com Dmitry Rybakov
            Reporter:
            bigbourin@gmail.com Adrien Jarthon
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: