Uploaded image for project: 'Ruby Driver'
  1. Ruby Driver
  2. RUBY-1030

Mongo Driver throws "No server is available matching preference" after replica set primary went down

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 12_01_17
    • Affects Version/s: 2.0.6
    • Component/s: Replica Set
    • Labels:
    • Environment:
      JRuby 1.7.20
      MongoDB Server 2.4
      Mongo Ruby Driver 2.0.6
      Linux prod-xxx-db6 3.13.0-46-generic #79-Ubuntu SMP Tue Mar 10 20:06:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


      We had a network/hardware issue with one of the replica set primary (hosted with AWS), and that caused the primary to switch to one of the secondary servers. However, our application went down, cause the driver did NOT update its cluster and was throwing this error

      "No server is available matching preference: #"prod", :max_pool_size=>200, :connect=>:replica_set, :write=>{:w=>1}}, @server_selection_timeout=30>"

      We use the following Mongo::Client options on the apps:
      max_pool_size: 200
      connect: replica_set

      {w: 1}

      hosts: ['prod-xxx-db4:27017', 'prod-xxx-db6:27017', 'prod-xxx-db7:27017', 'prod-xxx-db8:27017']

      Mongo Server 2.4
      Mongo Ruby Driver 2.0.6
      JRuby 1.7.20 (1.9 mode)

      Event Timeline:
      Aug 28 11:33 AM : The original primary (prod-xxx-db7) went inaccessible (not able to connect to it)
      Aug 28 11:34-11:36 AM : Mongo Connection timeout errors on App servers
      Aug 28 11:36 AM: prod-xxx-db6 was elected as primary
      Aug 28 11:36-11:55 AM: Mongo connection errors ("no server is available") on app servers for queries to the 'prod' replica set
      Aug 28 11:52 AM - App server restart fixed the errors.

      Is there any known issue which causes this behavior? If not, please review and let us know if you need more info. Thanks!

            emily.stolfo Emily Stolfo
            sathya@poshmark.com Sathya Sundaram
            0 Vote for this issue
            3 Start watching this issue