-
Type: Bug
-
Resolution: Fixed
-
Priority: Critical - P2
-
Affects Version/s: 2.19.0
-
Component/s: Connections
-
None
Hi,
I just upgraded from 2.18.2 to 2.19.0 3 days ago, and very quickly noticed a surge of Mongo::Error::ConnectionCheckOutTimeout errors. I let it run for a couple days as I wasn't sure what was the cause yet and suspected poor server performance, but today I tried to revert the mongo bump (at 14:47) and the errors immediately stopped (last one at 14:43), confirming IMO that some change in 2.19.0 is causing it. I had 2345 occurences in total across the 3 days that 2.19.0 have been in production.
Here is the exception in more details:
I'm running Sidekiq with 80 threads, so I configured my Mongoid pool to 100:
---
:concurrency: 80
:queues:
- mailers
- default
- low
production: clients: default: database: updown hosts: - db1.updn.io - db2.updn.io - db3.updn.io options: user: updown password: <%= Rails.application.credentials.dig(:mongodb, :production) %> compressors: [zstd, zlib] retry_writes: true max_read_retries: 3 read: mode: :primary_preferred max_pool_size: 100 wait_queue_timeout: 5 socket_timeout: 60
Under these condition there is no reason to hit this exception ever, and as a matter of fast I never did for more than 2 years!
I tried to look at the code diff but it's enormous so there's way I'll be able to find the reason on my own. Do you have any idea what code change in 2.19.0 could be causing this?