I was trying to configure one server of a RS with no votes so I could add another member, but after calling rs.reconfig() the primary server died.

Here's a transcript of the shell session:

PRIMARY> c=rs.conf()
{
"_id" : "share",
"version" : 43,
"members" : [

{ "_id" : 0, "host" : "sharemdb01a:27017" }

,

{ "_id" : 1, "host" : "sharemdb01b:27017" }

,

{ "_id" : 3, "host" : "arbitermdb02:29001", "arbiterOnly" : true }

,
{
"_id" : 4,
"host" : "sharemdb01d:27017",
"priority" : 0,
"slaveDelay" : 28800,
"initialSync" :

{ "name" : "sharemdb01b:27017" }

},

{ "_id" : 5, "host" : "mongobrick02:27017" }

,
{
"_id" : 6,
"host" : "sharemdb02a:27017",
"initialSync" :

{ "name" : "sharemdb01b:27017" }

},
{
"_id" : 8,
"host" : "sharemdb02c:27017",
"initialSync" :

{ "name" : "sharemdb01a:27017" }

}
]
}
PRIMARY> c.members[0]

{ "_id" : 0, "host" : "sharemdb01a:27017" }

PRIMARY> c.members[0].votes=0
0
PRIMARY> c
{
"_id" : "share",
"version" : 43,
"members" : [

{ "_id" : 0, "host" : "sharemdb01a:27017", "votes" : 0 }

,

{ "_id" : 1, "host" : "sharemdb01b:27017" }

,

{ "_id" : 3, "host" : "arbitermdb02:29001", "arbiterOnly" : true }

,
{
"_id" : 4,
"host" : "sharemdb01d:27017",
"priority" : 0,
"slaveDelay" : 28800,
"initialSync" :

{ "name" : "sharemdb01b:27017" }

},

{ "_id" : 5, "host" : "mongobrick02:27017" }

,
{
"_id" : 6,
"host" : "sharemdb02a:27017",
"initialSync" :

{ "name" : "sharemdb01b:27017" }

},
{
"_id" : 8,
"host" : "sharemdb02c:27017",
"initialSync" :

{ "name" : "sharemdb01a:27017" }

}
]
}
PRIMARY> rs.reconfig(c)
Mon Oct 17 16:31:00 DBClientCursor::init call() failed
Mon Oct 17 16:31:00 query failed : admin.$cmd { replSetReconfig: { _id: "share", version: 44, members: [

{ _id: 0, host: "sharemdb01a:27017", votes: 0.0 }

,

{ _id: 1, host: "sharemdb01b:27017" }

,

{ _id: 3, host: "arbitermdb02:29001", arbiterOnly: true }

, { _id: 4, host: "sharemdb01d:27017", priority: 0.0, slaveDelay: 28800, initialSync:

{ name: "sharemdb01b:27017" }

},

{ _id: 5, host: "mongobrick02:27017" }

, { _id: 6, host: "sharemdb02a:27017", initialSync:

{ name: "sharemdb01b:27017" }

}, { _id: 8, host: "sharemdb02c:27017", initialSync:

{ name: "sharemdb01a:27017" }

} ] } } to: sharemdb02a:27017
Mon Oct 17 16:31:00 trying reconnect to sharemdb02a:27017
Mon Oct 17 16:31:00 reconnect sharemdb02a:27017 ok
reconnected to server after rs command (which is normal)

Mon Oct 17 16:31:01 DBClientCursor::init call() failed
>
Mon Oct 17 16:31:02 trying reconnect to sharemdb02a:27017
Mon Oct 17 16:31:02 reconnect sharemdb02a:27017 failed couldn't connect to server sharemdb02a:27017
>
>

The segment corresponding segment in the logs:

Mon Oct 17 16:31:01 [conn24542] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "share", version: 44, members: [

{ _id: 0, host: "sharemdb01a:27017", vote s: 0.0 }

,

{ _id: 1, host: "sharemdb01b:27017" }

,

{ _id: 3, host: "arbitermdb02:29001", arbiterOnly: true }

, { _id: 4, host: "sharemdb01d:27017", priority: 0.0, slaveDelay
: 28800, initialSync:

{ name: "sharemdb01b:27017" }

},

{ _id: 5, host: "mongobrick02:27017" }

, { _id: 6, host: "sharemdb02a:27017", initialSync:

{ name: "sharemdb01b:2701 7" }

}, { _id: 8, host: "sharemdb02c:27017", initialSync:

{ name: "sharemdb01a:27017" }

} ] } } reslen:53 715ms
Mon Oct 17 16:31:02 [conn24437] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24542] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01d:27017 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [conn24402] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24405] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24452] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01a:27017 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb01b:27017 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [conn24491] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24414] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24415] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info arbitermdb02:29001 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [conn24497] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info mongobrick02:27017 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [conn24490] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [ReplSetHealthPollTask] replSet info sharemdb02c:27017 is down (or slow to respond): socket exception
Mon Oct 17 16:31:02 [conn24502] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24535] SocketException in connThread, closing client connection
Mon Oct 17 16:31:02 [conn24441] SocketException in connThread, closing client connection
/opt/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54ef1e]
/opt/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x56056d]
/opt/mongodb/bin/mongod(ZN5mongo11ReplSetImpl17_fillIsMasterHostEPKNS_6MemberERSt6vectorISsSaISsEES7_S7+0x213) [0x6c0173]
/opt/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl13_fillIsMasterERNS_14BSONObjBuilderE+0x27a) [0x6c04fa]
/opt/mongodb/bin/mongod(_ZN5mongo21appendReplicationInfoERNS_14BSONObjBuilderEbi+0x44) [0x69d704]
/opt/mongodb/bin/mongod(_ZN5mongo11CmdIsMaster3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x88) [0x6b6c18]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
/opt/mongodb/bin/mongod [0x75591c]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
/lib64/libpthread.so.0 [0x36f66064a7]
/lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
/opt/mongodb/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54ef1e]
/opt/mongodb/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfd) [0x56056d]
/opt/mongodb/bin/mongod(ZN5mongo11ReplSetImpl17_fillIsMasterHostEPKNS_6MemberERSt6vectorISsSaISsEES7_S7+0x213) [0x6c0173]
/opt/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl13_fillIsMasterERNS_14BSONObjBuilderE+0x27a) [0x6c04fa]
/opt/mongodb/bin/mongod(_ZN5mongo21appendReplicationInfoERNS_14BSONObjBuilderEbi+0x44) [0x69d704]
/opt/mongodb/bin/mongod(_ZN5mongo11CmdIsMaster3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x88) [0x6b6c18]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
/opt/mongodb/bin/mongod [0x75591c]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
/lib64/libpthread.so.0 [0x36f66064a7]
/lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
Mon Oct 17 16:31:02 Backtrace:
0x8ade79 0x8ae450 0x36f660e7c0 0x6d9cc2 0x6d2282 0x7e08df 0x7e2126 0x645c35 0x64c30a 0x75591c 0x75d378 0x8aecbe 0x36f66064a7 0x36f5ad3c2d
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8ade79]
/opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0x8ae450]
/lib64/libpthread.so.0 [0x36f660e7c0]
/opt/mongodb/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x72) [0x6d9cc2]
/opt/mongodb/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x122) [0x6d2282]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]

/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
/opt/mongodb/bin/mongod [0x75591c]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
/lib64/libpthread.so.0 [0x36f66064a7]
/lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]
Mon Oct 17 16:31:02 Backtrace:
0x8ade79 0x8ae450 0x36f660e7c0 0x6d9cc2 0x6d2282 0x7e08df 0x7e2126 0x645c35 0x64c30a 0x75591c 0x75d378 0x8aecbe 0x36f66064a7 0x36f5ad3c2d
/opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x8ade79]
/opt/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0x8ae450]
/lib64/libpthread.so.0 [0x36f660e7c0]
/opt/mongodb/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x72) [0x6d9cc2]
/opt/mongodb/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x122) [0x6d2282]
/opt/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x41f) [0x7e08df]
/opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x616) [0x7e2126]
/opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x645c35]
/opt/mongodb/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x32da) [0x64c30a]
/opt/mongodb/bin/mongod [0x75591c]
/opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x75d378]
/opt/mongodb/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8aecbe]
/lib64/libpthread.so.0 [0x36f66064a7]
/lib64/libc.so.6(clone+0x6d) [0x36f5ad3c2d]

Mon Oct 17 16:31:02 dbexit:

full logfile attached.