Uploaded image for project: 'Mongoid'
  1. Mongoid
  2. MONGOID-4253

Connections increase beyond manageable amount with sidekiq 3.5.4

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • 12_01_17
    • Affects Version/s: 5.1.0, 5.1.1
    • Component/s: None
    • Labels:
      None
    • Environment:
      Heroku, ruby 2.2.3, rails 4.1.9, sidekiq 3.5.4, puma 2.15.3

      We switched our production app from mongoid <5 to mongoid 5.1.0 a few weeks ago. It wound up crashing our provider's mongoDB servers over and over again, and we had to switch back. (Unfortunately, we only found this issue when in production, which is hard to duplicate as it requires tens to hundreds of jobs per second executing over multiple sidekiq processes, plus a user-facing site with moderate traffic, to be present before it shows itself.)

      Our database provider (Compose) put it this way:

      "The ingress points are being connection stormed. We have a 5,000 open connection limit on each ingress point. There appears to be a problem with connections increasing and not being closed. It appears to be a rising level of connections that kills the first `mongos` and then the second one is used... where the same problem eventually happens."

      I watched our connections myself as the servers continued to go down and sometimes restart, and they would climb close to 1000 and then rapidly increase, all over the span of a couple of hours. We had our mongoid connection pool size set to 125, so this should not have been possible. We don't have any issues with the same version of sidekiq running on mongoid 4.0.2, the only code change was upgrading mongoid.

      I realize this isn't a lot to go on, but I wanted to open something so you would be aware there was something up here. All I can guess is that there's a connection pooling issue, and the usage of sidekiq exacerbated this and made it a very big problem. For now, we're sticking with 4.0.2.

            Assignee:
            emily.stolfo Emily Stolfo
            Reporter:
            danarnold_ Dan Arnold
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: