Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-4091

rs.reconfig(...) on 1.8.4_rc0 causes server to exit

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Duplicate
    • 1.8.4
    • None
    • Replication
    • None
    • linux x86_64
    • ALL

    Description

      I was trying to configure one server of a RS with no votes so I could add another member, but after calling rs.reconfig() the primary server died.

      Here's a transcript of the shell session:

      PRIMARY> c=rs.conf()
      {
      "_id" : "share",
      "version" : 43,
      "members" : [

      { "_id" : 0, "host" : "sharemdb01a:27017" }

      ,

      { "_id" : 1, "host" : "sharemdb01b:27017" }

      ,

      { "_id" : 3, "host" : "arbitermdb02:29001", "arbiterOnly" : true }

      ,
      {
      "_id" : 4,
      "host" : "sharemdb01d:27017",
      "priority" : 0,
      "slaveDelay" : 28800,
      "initialSync" :

      { "name" : "sharemdb01b:27017" }

      },

      { "_id" : 5, "host" : "mongobrick02:27017" }

      ,
      {
      "_id" : 6,
      "host" : "sharemdb02a:27017",
      "initialSync" :

      { "name" : "sharemdb01b:27017" }

      },
      {
      "_id" : 8,
      "host" : "sharemdb02c:27017",
      "initialSync" :

      { "name" : "sharemdb01a:27017" }

      }
      ]
      }
      PRIMARY> c.members[0]

      { "_id" : 0, "host" : "sharemdb01a:27017" }

      PRIMARY> c.members[0].votes=0
      0
      PRIMARY> c
      {
      "_id" : "share",
      "version" : 43,
      "members" : [

      { "_id" : 0, "host" : "sharemdb01a:27017", "votes" : 0 }

      ,

      { "_id" : 1, "host" : "sharemdb01b:27017" }

      ,

      { "_id" : 3, "host" : "arbitermdb02:29001", "arbiterOnly" : true }

      ,
      {
      "_id" : 4,
      "host" : "sharemdb01d:27017",
      "priority" : 0,
      "slaveDelay" : 28800,
      "initialSync" :

      { "name" : "sharemdb01b:27017" }

      },

      { "_id" : 5, "host" : "mongobrick02:27017" }

      ,
      {
      "_id" : 6,
      "host" : "sharemdb02a:27017",
      "initialSync" :

      { "name" : "sharemdb01b:27017" }

      },
      {
      "_id" : 8,
      "host" : "sharemdb02c:27017",
      "initialSync" :

      { "name" : "sharemdb01a:27017" }

      }
      ]
      }
      PRIMARY> rs.reconfig(c)
      Mon Oct 17 16:31:00 DBClientCursor::init call() failed
      Mon Oct 17 16:31:00 query failed : admin.$cmd { replSetReconfig: { _id: "share", version: 44, members: [

      { _id: 0, host: "sharemdb01a:27017", votes: 0.0 }

      ,

      { _id: 1, host: "sharemdb01b:27017" }

      ,

      { _id: 3, host: "arbitermdb02:29001", arbiterOnly: true }

      , { _id: 4, host: "sharemdb01d:27017", priority: 0.0, slaveDelay: 28800, initialSync:

      { name: "sharemdb01b:27017" }

      },

      { _id: 5, host: "mongobrick02:27017" }

      , { _id: 6, host: "sharemdb02a:27017", initialSync:

      { name: "sharemdb01b:27017" }

      }, { _id: 8, host: "sharemdb02c:27017", initialSync:

      { name: "sharemdb01a:27017" }

      } ] } } to: sharemdb02a:27017
      Mon Oct 17 16:31:00 trying reconnect to sharemdb02a:27017
      Mon Oct 17 16:31:00 reconnect sharemdb02a:27017 ok
      reconnected to server after rs command (which is normal)

      Mon Oct 17 16:31:01 DBClientCursor::init call() failed
      >
      Mon Oct 17 16:31:02 trying reconnect to sharemdb02a:27017
      Mon Oct 17 16:31:02 reconnect sharemdb02a:27017 failed couldn't connect to server sharemdb02a:27017
      >
      >

      The segment corresponding segment in the logs:

      Mon Oct 17 16:31:01 [conn24542] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "share", version: 44, members: [

      { _id: 0, host: "sharemdb01a:27017", vote s: 0.0 }

      ,

      { _id: 1, host: "sharemdb01b:27017" }

      ,

      { _id: 3, host: "arbitermdb02:29001", arbiterOnly: true }

      , { _id: 4, host: "sharemdb01d:27017", priority: 0.0, slaveDelay
      : 28800, initialSync:

      { name: "sharemdb01b:27017" }

      },

      { _id: 5, host: "mongobrick02:27017" }

      , { _id: 6, host: "sharemdb02a:27017", initialSync:

      { name: "sharemdb01b:2701 7" }

      }, { _id: 8, host: "sharemdb02c:27017", initialSync:

      { name: "sharemdb01a:27017" }

      } ] } } reslen:53 715ms
      Mon Oct 17 16:31:02 [conn24437] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24542] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01d:27017 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [conn24402] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24405] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24452] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01a:27017 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01b:27017 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [conn24491] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24414] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24415] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info arbitermdb02:29001 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [conn24497] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info mongobrick02:27017 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [conn24490] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb02c:27017 is down (or slow to respond): socket exception
      Mon Oct 17 16:31:02 [conn24502] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24535] SocketException in connThread, closing client connection
      Mon Oct 17 16:31:02 [conn24441] SocketException in connThread, closing client connection
      /opt/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54ef1e]
      /opt/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x56056d]
      /opt/mongodb/bin/mongod(ZN5mongo11ReplSetImpl17_fillIsMasterHostEPKNS_6MemberERSt6vectorISsSaISsEES7_S7+0x213) [0x6c0173]
      /opt/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl13_fillIsMasterERNS_14BSONObjBuilderE+0x27a) [0x6c04fa]
      /opt/mongodb/bin/mongod(_ZN5mongo21appendReplicationInfoERNS_14BSONObjBuilderEbi+0x44) [0x69d704]
      /opt/mongodb/bin/mongod(_ZN5mongo11CmdIsMaster3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x88) [0x6b6c18]
      /opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
      /opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
      /opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
      /opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
      /opt/mongodb/bin/mongod [0x75591c]
      /opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
      /opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
      /lib64/libpthread.so.0 [0x36f66064a7]
      /lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
      /opt/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54ef1e]
      /opt/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x56056d]
      /opt/mongodb/bin/mongod(ZN5mongo11ReplSetImpl17_fillIsMasterHostEPKNS_6MemberERSt6vectorISsSaISsEES7_S7+0x213) [0x6c0173]
      /opt/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl13_fillIsMasterERNS_14BSONObjBuilderE+0x27a) [0x6c04fa]
      /opt/mongodb/bin/mongod(_ZN5mongo21appendReplicationInfoERNS_14BSONObjBuilderEbi+0x44) [0x69d704]
      /opt/mongodb/bin/mongod(_ZN5mongo11CmdIsMaster3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x88) [0x6b6c18]
      /opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
      /opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
      /opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
      /opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
      /opt/mongodb/bin/mongod [0x75591c]
      /opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
      /opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
      /lib64/libpthread.so.0 [0x36f66064a7]
      /lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
      Mon Oct 17 16:31:02 Backtrace:
      0x8ade79 0x8ae450 0x36f660e7c0 0x6d9cc2 0x6d2282 0x7e08df 0x7e2126 0x645c35 0x64c30a 0x75591c 0x75d378 0x8aecbe 0x36f66064a7 0x36f5ad3c2d
      /opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8ade79]
      /opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0x8ae450]
      /lib64/libpthread.so.0 [0x36f660e7c0]
      /opt/mongodb/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x72) [0x6d9cc2]
      /opt/mongodb/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x122) [0x6d2282]
      /opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
      /opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]

      /opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
      /opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
      /opt/mongodb/bin/mongod [0x75591c]
      /opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
      /opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
      /lib64/libpthread.so.0 [0x36f66064a7]
      /lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
      Mon Oct 17 16:31:02 Backtrace:
      0x8ade79 0x8ae450 0x36f660e7c0 0x6d9cc2 0x6d2282 0x7e08df 0x7e2126 0x645c35 0x64c30a 0x75591c 0x75d378 0x8aecbe 0x36f66064a7 0x36f5ad3c2d
      /opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8ade79]
      /opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0x8ae450]
      /lib64/libpthread.so.0 [0x36f660e7c0]
      /opt/mongodb/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x72) [0x6d9cc2]
      /opt/mongodb/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x122) [0x6d2282]
      /opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
      /opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
      /opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
      /opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
      /opt/mongodb/bin/mongod [0x75591c]
      /opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
      /opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
      /lib64/libpthread.so.0 [0x36f66064a7]
      /lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]

      Mon Oct 17 16:31:02 dbexit:

      full logfile attached.

      Attachments

        Issue Links

          Activity

            People

              kristina Kristina Chodorow (Inactive)
              mpatenaude Mitch Patenaude
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: