Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3133

mongos crash on replicaset failover

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.8.2
    • Affects Version/s: 1.8.1
    • Component/s: None
    • ALL

      we are running a sharded setup with 2 shards and each shard is represented by a replicaset of 2 replicas and 1 arbiter. on the primary for 1 shard, we issued the rs.stepDown() command so that we could perform maintenance on the box. for 5s+, both replicas in the set reported as secondary nodes. subsequently, the mongos processed crashed with the following trace:

      Received signal 6
      Backtrace: 0x52e235 0x7f7d51221af0 0x7f7d51221a75 0x7f7d512255c0 0x7f7d5121a941 0x5517c3 0x552208 0x54a10c 0x53fcca 0x577eae 0x5789dc 0x69dae1 0x69ec38 0x7f7d51d259ca 0x7f7d512d470d
      /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib/libc.so.6(+0x33af0)[0x7f7d51221af0]
      /lib/libc.so.6(gsignal+0x35)[0x7f7d51221a75]
      /lib/libc.so.6(abort+0x180)[0x7f7d512255c0]
      /lib/libc.so.6(__assert_fail+0xf1)[0x7f7d5121a941]
      /usr/bin/mongos(_ZN5mongo18DBClientReplicaSet11checkMasterEv+0x4b3)[0x5517c3]
      /usr/bin/mongos(_ZN5mongo18DBClientReplicaSet7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x128)[0x552208]
      /usr/bin/mongos(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x8c)[0x54a10c]
      /usr/bin/mongos(ZN5mongo20DBClientWithCommands13simpleCommandERKSsPNS_7BSONObjES2+0x12a)[0x53fcca]
      /usr/bin/mongos(_ZN5mongo17ClientConnections7releaseERKSsPNS_12DBClientBaseE+0x10e)[0x577eae]
      /usr/bin/mongos(_ZN5boost19thread_specific_ptrIN5mongo17ClientConnectionsEE11delete_dataclEPv+0xac)[0x5789dc]
      /usr/bin/mongos(tls_destructor+0xb1)[0x69dae1]
      /usr/bin/mongos(thread_proxy+0x88)[0x69ec38]
      /lib/libpthread.so.0(+0x69ca)[0x7f7d51d259ca]
      /lib/libc.so.6(clone+0x6d)[0x7f7d512d470d]
      ===
      Received signal 11
      Backtrace: 0x52e235 0x7f7d51221af0 0x5394ba 0x5595a5 0x53ae5a 0x54af51 0x546afc 0x54a10c 0x53bfd5 0x54fd78 0x5507fd 0x5510a2 0x55134d 0x552208 0x54a10c 0x53fcca 0x577eae 0x5789dc 0x69f3c1 0x573b22
      /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib/libc.so.6(+0x33af0)[0x7f7d51221af0]
      /usr/bin/mongos(_ZN5mongo15assembleRequestERKSsNS_7BSONObjEiiPKS2_iRNS_7MessageE+0xea)[0x5394ba]
      /usr/bin/mongos(_ZN5mongo14DBClientCursor4initEv+0x105)[0x5595a5]
      /usr/bin/mongos(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x32a)[0x53ae5a]
      /usr/bin/mongos(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1)[0x54af51]
      /usr/bin/mongos(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x7c)[0x546afc]
      /usr/bin/mongos(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x8c)[0x54a10c]
      /usr/bin/mongos(_ZN5mongo20DBClientWithCommands8isMasterERbPNS_7BSONObjE+0x75)[0x53bfd5]
      /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor16_checkConnectionEPNS_18DBClientConnectionERSsb+0x88)[0x54fd78]
      /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor6_checkEv+0x14d)[0x5507fd]
      /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor9getMasterEv+0x72)[0x5510a2]
      /usr/bin/mongos(_ZN5mongo18DBClientReplicaSet11checkMasterEv+0x3d)[0x55134d]
      /usr/bin/mongos(_ZN5mongo18DBClientReplicaSet7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x128)[0x552208]
      /usr/bin/mongos(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x8c)[0x54a10c]
      /usr/bin/mongos(ZN5mongo20DBClientWithCommands13simpleCommandERKSsPNS_7BSONObjES2+0x12a)[0x53fcca]
      /usr/bin/mongos(_ZN5mongo17ClientConnections7releaseERKSsPNS_12DBClientBaseE+0x10e)[0x577eae]
      /usr/bin/mongos(_ZN5boost19thread_specific_ptrIN5mongo17ClientConnectionsEE11delete_dataclEPv+0xac)[0x5789dc]
      /usr/bin/mongos(_ZN5boost6detail12set_tss_dataEPKvNS_10shared_ptrINS0_20tss_cleanup_functionEEEPvb+0x151)[0x69f3c1]
      /usr/bin/mongos[0x573b22]
      ===

            Assignee:
            Unassigned Unassigned
            Reporter:
            wayne530 Y. Wayne Huang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: