Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26701

MongoS stalls when it cannot access one of the CSRS server

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.2.5
    • Fix Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Operating System:
      ALL

      Description

      Running a sharded cluster using MongoDb3.2.5 with WiredTiger and of late, mongos on some machine just stalls when it cannot access one of the CSRS server.

      CSRS is a 3 node replicaset and only a couple of mongos processes get stuck in this scenario - we can ignore the reason for one CSRS server not being available for this particular problem

      Here is the mogos log snippet when this happened - the mongos tried to switch the order of the CSRS servers and then after a couple of attempts, it just hung. Any queries using that mongos process would not return.

      2016-10-19T09:02:30.381-0400 I SHARDING [Balancer] distributed lock 'balancer' acquired for 'doing balance round', ts : 58076ee6087c1d793ad3b986
      2016-10-19T09:02:32.657-0400 I SHARDING [Balancer] distributed lock with ts: 58076ee6087c1d793ad3b986' unlocked.
      2016-10-19T09:03:47.065-0400 I SHARDING [Balancer] distributed lock 'balancer' acquired for 'doing balance round', ts : 58076f33087c1d793ad3b98d
      2016-10-19T09:03:49.341-0400 I SHARDING [Balancer] distributed lock with ts: 58076f33087c1d793ad3b98d' unlocked.
      2016-10-19T09:05:28.947-0400 I NETWORK  [ReplicaSetMonitorWatcher] changing hosts to csReplSet/mongoconfigserver1:29102,mongoconfigserver3:29102,mongoconfigserver2:29102 from csReplSet/mongoconfigserver1:29102,mongoconfigserver2:29102
      2016-10-19T09:05:28.947-0400 I SHARDING [ReplicaSetMonitorWatcher] Updating config server connection string to: csReplSet/mongoconfigserver1:29102,mongoconfigserver3:29102,mongoconfigserver2:29102
      2016-10-19T09:05:28.947-0400 I SHARDING [ReplicaSetMonitorWatcher] Updating ShardRegistry connection string for shard config from: csReplSet/mongoconfigserver1:29102,mongoconfigserver2:29102 to: csReplSet/mongoconfigserver1:29102,mongoconfigserver3:29102,mongoconfigserver2:29102
      2016-10-19T09:06:15.520-0400 W SHARDING [Balancer] ExceededTimeLimit: Couldn't get a connection within the time limit
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              darshan.shah@interactivedata.com Darshan Shah
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: