Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-1533

Replica set SECONDARY crash when terminating other PRIMARY and SECONDARY

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.5.7
    • Component/s: None
    • Labels:
      None
    • Linux

      I am unable to reproduce this again and am not totally sure of the circumstances leading to it, but there may be some useful information in the backtrace.

      I had 3 members of replica set. I terminate the primary and then immediately after one of the secondaries. The third secondary crashed on its own.

      PRIMARY

      mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-89-F1.compute-1.internal:27018 --rest

      Sun Aug 1 05:16:33 [conn9] getmore local.oplog.rs cid:5805707532161602891 getMore: { ts:

      { $gte: new Date(5500358045240131585) }

      } bytes:20 nreturned:0 3007ms
      Sun Aug 1 05:16:45 [conn9] getmore local.oplog.rs cid:5805707532161602891 getMore: { ts:

      { $gte: new Date(5500358045240131585) }

      } bytes:20 nreturned:0 3003ms
      Sun Aug 1 05:16:46 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
      Sun Aug 1 05:16:46 [interruptThread] now exiting
      Sun Aug 1 05:16:46 dbexit:

      Sun Aug 1 05:16:46 [interruptThread] shutdown: going to close listening sockets...
      Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 9
      Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 10
      Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 11
      Sun Aug 1 05:16:46 [interruptThread] going to close listening socket: 12
      Sun Aug 1 05:16:46 [interruptThread] shutdown: going to flush oplog...
      Sun Aug 1 05:16:46 [interruptThread] shutdown: going to close sockets...
      Sun Aug 1 05:16:46 [interruptThread] shutdown: waiting for fs preallocator...
      Sun Aug 1 05:16:46 [interruptThread] shutdown: closing all files...
      Sun Aug 1 05:16:46 closeAllFiles() finished

      Sun Aug 1 05:16:46 [interruptThread] shutdown: removing fs lock...
      Sun Aug 1 05:16:46 [conn1] end connection 127.0.0.1:43135
      Sun Aug 1 05:16:46 [conn4] end connection 10.208.46.160:43621
      Sun Aug 1 05:16:46 dbexit: really exiting now

      SECONDARY 1

      mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-29-52.compute-1.internal:27018 --rest

      Sun Aug 1 05:17:57 [conn5] getmore local.oplog.rs cid:5040576339790303823 getMore: { ts:

      { $gte: new Date(5500358045240131585) }

      } bytes:20 nreturned:0 3007ms
      Sun Aug 1 05:17:58 [conn6] getmore local.oplog.rs cid:2586884600444785463 getMore: { ts:

      { $gte: new Date(5500358045240131585) }

      } bytes:20 nreturned:0 3007ms
      Sun Aug 1 05:18:03 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
      Sun Aug 1 05:18:03 [interruptThread] now exiting
      Sun Aug 1 05:18:03 dbexit:

      Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close listening sockets...
      Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 10
      Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 11
      Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 12
      Sun Aug 1 05:18:03 [interruptThread] going to close listening socket: 13
      Sun Aug 1 05:18:03 [interruptThread] shutdown: going to flush oplog...
      Sun Aug 1 05:18:03 [interruptThread] shutdown: going to close sockets...
      Sun Aug 1 05:18:03 [interruptThread] shutdown: waiting for fs preallocator...
      Sun Aug 1 05:18:03 [interruptThread] shutdown: closing all files...
      Sun Aug 1 05:18:03 [conn2] end connection 127.0.0.1:48610
      Sun Aug 1 05:18:03 closeAllFiles() finished

      Sun Aug 1 05:18:03 [interruptThread] shutdown: removing fs lock...
      Sun Aug 1 05:18:03 dbexit: really exiting now

      SECONDARY 2

      mongodb-linux-x86_64-1.5.7/bin/mongod --shardsvr --dbpath data/ --replSet set1/domU-12-31-39-06-29-52.compute-1.internal:27018 --rest

      Sun Aug 1 04:38:36 [rs_sync] replSet SECONDARY
      Sun Aug 1 05:16:47 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
      Sun Aug 1 05:16:47 [conn2] end connection 10.208.46.160:53683
      Sun Aug 1 05:16:48 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now down (or slow to respond)
      Sun Aug 1 05:16:48 [conn3] replSet info voting yea for 1
      Sun Aug 1 05:16:49 [rs Manager] replSet TEMP not voting yea for 2
      Sun Aug 1 05:16:49 [rs Manager] replSet TEMP voted for 1 1 secs ago
      Sun Aug 1 05:16:49 [rs Manager] replSet not trying to elect self as responded yea to someone else recently
      Sun Aug 1 05:16:56 [ReplSetHealthPollTask] replSet info domU-12-31-39-06-29-52.compute-1.internal:27018 is now up
      Sun Aug 1 05:16:57 [initandlisten] connection accepted from 10.208.46.160:50868 #4
      Sun Aug 1 05:16:58 [rs_sync] replSet SECONDARY
      Sun Aug 1 05:18:03 [rs_sync] replSet syncThread: 10278 dbclient error communicating with server
      Sun Aug 1 05:18:03 [conn3] end connection 10.208.142.3:45480
      Sun Aug 1 05:18:03 [conn4] replSet info voting yea for 0
      Sun Aug 1 05:18:04 [rs Manager] should never be called?
      Sun Aug 1 05:18:04 [rs Manager] Assertion failure false db/repl/manager.cpp 55
      0x5313c3 0x53db81 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb3) [0x5313c3]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x53db81]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
      mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
      /lib64/libpthread.so.0 [0x2aaaaaccd407]
      /lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
      Sun Aug 1 05:18:04 terminate() called, printing stack:

      0x801ab0 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x50) [0x801ab0]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
      mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
      /lib64/libpthread.so.0 [0x2aaaaaccd407]
      /lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]
      Sun Aug 1 05:18:04 Got signal: 6 (Aborted).

      Sun Aug 1 05:18:04 Backtrace:
      0x8025c9 0x2aaaab6a4f30 0x2aaaab6a4ec5 0x2aaaab6a6970 0x801b6f 0x2aaaaaf9ee76 0x2aaaaaf9eea3 0x2aaaaaf9ef8a 0x53ddfc 0x67ca7b 0x526a08 0x818340 0x2aaaaaccd407 0x2aaaab748b0d
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8025c9]
      /lib64/libc.so.6 [0x2aaaab6a4f30]
      /lib64/libc.so.6(gsignal+0x35) [0x2aaaab6a4ec5]
      /lib64/libc.so.6(abort+0x110) [0x2aaaab6a6970]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo11myterminateEv+0x10f) [0x801b6f]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9ee76]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9eea3]
      /usr/lib64/libstdc++.so.6 [0x2aaaaaf9ef8a]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x38c) [0x53ddfc]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo7ManagerD0Ev+0x9b) [0x67ca7b]
      mongodb-linux-x86_64-1.5.7/bin/mongod(_ZN5mongo13BackgroundJob3thrEv+0xa8) [0x526a08]
      mongodb-linux-x86_64-1.5.7/bin/mongod(thread_proxy+0x80) [0x818340]
      /lib64/libpthread.so.0 [0x2aaaaaccd407]
      /lib64/libc.so.6(clone+0x6d) [0x2aaaab748b0d]

      Sun Aug 1 05:18:04 dbexit:

      Sun Aug 1 05:18:04 [rs Manager] shutdown: going to close listening sockets...
      Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 10
      Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 11
      Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 12
      Sun Aug 1 05:18:04 [rs Manager] going to close listening socket: 13
      Sun Aug 1 05:18:04 [rs Manager] shutdown: going to flush oplog...
      Sun Aug 1 05:18:04 [rs Manager] shutdown: going to close sockets...
      Sun Aug 1 05:18:04 [rs Manager] shutdown: waiting for fs preallocator...
      Sun Aug 1 05:18:04 [rs Manager] shutdown: closing all files...
      Sun Aug 1 05:18:04 closeAllFiles() finished

      Sun Aug 1 05:18:04 [rs Manager] shutdown: removing fs lock...
      Sun Aug 1 05:18:04 dbexit: really exiting now

            Assignee:
            dwight@mongodb.com Dwight Merriman
            Reporter:
            boxedice David Mytton
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: