[SERVER-10043] rs.reconfig fails with exception: assertion src/mongo/db/repl/rs_initiate.cpp:125 Created: 27/Jun/13  Updated: 10/Dec/14  Resolved: 02/Jul/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.4.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nic Cottrell (Personal) Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL5


Operating System: ALL
Participants:

 Description   

rs.config() gives

{
	"_id" : "sprawk",
	"version" : 1,
	"members" : [
		{
			"_id" : 0,
			"host" : "sture.transmachina.net:27017"
		},
		{
			"_id" : 1,
			"host" : "sprawk.member0.mongolayer.com:27017",
			"priority" : 0
		},
		{
			"_id" : 2,
			"host" : "oden.transmachina.net:27017",
			"arbiterOnly" : true
		}
	]
}

but sprawk.member0.mongolayer.com no longer exists. We have a new node on our own server. So I run rs.reconfig

rs.reconfig( 
     {"_id": "sprawk", "version" : 2, "members": [ 
          {"_id": 0, "host": "sture.transmachina.net:27017", priority: 2}, 
          {"_id": 1, "host": "haga.transmachina.net:27017", priority: 1}, 
          {"_id": 2, "host": "oden.transmachina.net:27017", "arbiterOnly": true} 
     ] } );

but get

{
	"errmsg" : "exception: assertion src/mongo/db/repl/rs_initiate.cpp:125",
	"code" : 0,
	"ok" : 0
}



 Comments   
Comment by Maga Napanga [ 28/Nov/14 ]

Hello!

I had the same issue (mongodb 2.6.1), but I noticed that the assertion happened only if I called rs.reconfig() while the changed host was down.
Bringing the rs member up before calling rs.reconfig() does work.

Below some shell and mongo logs:

--> this was the reconfigured rs

wsyrs:PRIMARY> cfg = rs.conf()
{
	"_id" : "wsyrs",
	"version" : 7,
	"members" : [
		{
			"_id" : 0,
			"host" : "rs0.wsy:27017"
		},
		{
			"_id" : 1,
			"host" : "rs1.wsy:27017"
		},
		{
			"_id" : 3,
			"host" : "mongo-arb-6ovf:27017",
			"arbiterOnly" : true
		},
		{
			"_id" : 4,
			"host" : "mongo-db-6m7q:27017",
			"priority" : 0
		},
		{
			"_id" : 5,
			"host" : "mongo-db-td6n:27017",
			"priority" : 0
		}
	]
}
wsyrs:PRIMARY> cfg.members[2].host = "arb.wsy:27017"
arb.wsy:27017
wsyrs:PRIMARY> rs.reconfig(cfg)
{
	"errmsg" : "exception: assertion src/mongo/db/repl/rs_initiate.cpp:138",
	"code" : 0,
	"ok" : 0
}
wsyrs:PRIMARY>

--> this is mongo log while failing to reconfig (new arbiter was down)

2014-11-28T14:21:15.627+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:21:15.627+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:21:16.657+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:21:16.698+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:21:16.698+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:21:18.037+0000 [conn24566] replSet replSetReconfig config object parses ok, 5 members specified
2014-11-28T14:21:18.039+0000 [conn24566] warning: Failed to connect to 10.128.229.18:27017, reason: errno:111 Connection refused
2014-11-28T14:21:18.039+0000 [conn24566] couldn't connect to arb.wsy:27017: couldn't connect to server arb.wsy:27017 (10.128.229.18) failed, connection attempt failed
2014-11-28T14:21:18.040+0000 [conn24566] warning: Failed to connect to 10.128.229.18:27017, reason: errno:111 Connection refused
2014-11-28T14:21:18.041+0000 [conn24566] replSet cmufcc requestHeartbeat arb.wsy:27017 : 9001 socket exception [CONNECT_ERROR] server [arb.wsy:27017 (10.128.229.18) failed] 
2014-11-28T14:21:18.041+0000 [conn24566] Assertion failure m->h().toString() == i->h.toString() src/mongo/db/repl/rs_initiate.cpp 138
2014-11-28T14:21:18.047+0000 [conn24566] 0x11c0e91 0x1163109 0x1146f4e 0xe87eb2 0xe54ba3 0xe554b0 0xa1e85a 0xa1f8ce 0xa21086 0xd4dae7 0xb97322 0xb99902 0x76b6af 0x117720b 0x7fb754230182 0x7fb75353530d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1163109]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x1146f4e]
 /usr/bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x2742) [0xe87eb2]
 /usr/bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x7a3) [0xe54ba3]
 /usr/bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x70) [0xe554b0]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa1e85a]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xd5e) [0xa1f8ce]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa21086]
 /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x2307) [0xd4dae7]
 /usr/bin/mongod() [0xb97322]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb99902]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b6af]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x117720b]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fb754230182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb75353530d]
2014-11-28T14:21:18.047+0000 [conn24566] replSet replSetReconfig exception: assertion src/mongo/db/repl/rs_initiate.cpp:138
2014-11-28T14:21:18.740+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:21:18.740+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:21:18.774+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:21:18.774+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed

--> bring new arbiter up, then reconfig again:

wsyrs:PRIMARY> cfg.members[2].host = "arb.wsy:27017"
arb.wsy:27017
wsyrs:PRIMARY> rs.reconfig(cfg)
2014-11-28T14:22:48.581+0000 DBClientCursor::init call() failed
2014-11-28T14:22:48.586+0000 trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2014-11-28T14:22:48.587+0000 reconnect 127.0.0.1:27017 (127.0.0.1) ok
reconnected to server after rs command (which is normal)
 
wsyrs:PRIMARY> 

--> this is mongo log while succeeding to reconfig (new arbiter was up)

2014-11-28T14:22:48.063+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:22:48.103+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:22:48.103+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:22:48.485+0000 [conn24566] replSet replSetReconfig config object parses ok, 5 members specified
2014-11-28T14:22:48.578+0000 [conn24566] replSet replSetReconfig [2]
2014-11-28T14:22:48.578+0000 [conn24566] replSet info saving a newer config version to local.system.replset: { _id: "wsyrs", version: 8, members: [ { _id: 0, host: "rs0.wsy:27017" }, { _id: 1, host: "rs1.wsy:27017" }, { _id: 3, host: "arb.wsy:27017", arbiterOnly: true }, { _id: 4, host: "mongo-db-6m7q:27017", priority: 0.0 }, { _id: 5, host: "mongo-db-td6n:27017", priority: 0.0 } ] }
2014-11-28T14:22:48.579+0000 [conn24566] replSet saveConfigLocally done
2014-11-28T14:22:48.580+0000 [conn24566] replSet relinquishing primary state
2014-11-28T14:22:48.580+0000 [conn24566] replSet SECONDARY
2014-11-28T14:22:48.580+0000 [conn24566] replSet closing client sockets after relinquishing primary
2014-11-28T14:22:48.580+0000 [conn24540] end connection 10.128.229.15:59530 (28 connections now open)
2014-11-28T14:22:48.580+0000 [conn22615] end connection 10.128.229.18:55955 (28 connections now open)
2014-11-28T14:22:48.581+0000 [conn24559] end connection w3.x.y.z:46655 (28 connections now open)
2014-11-28T14:22:48.582+0000 [initandlisten] connection accepted from 10.128.229.15:33396 #24654 (30 connections now open)
2014-11-28T14:22:48.581+0000 [conn23587] end connection 10.128.229.18:56730 (28 connections now open)
2014-11-28T14:22:48.583+0000 [conn24602] end connection w2.x.y.z:48989 (28 connections now open)
2014-11-28T14:22:48.584+0000 [conn23598] end connection 10.128.229.18:56748 (28 connections now open)
2014-11-28T14:22:48.584+0000 [conn23597] end connection 10.128.229.18:56745 (28 connections now open)
2014-11-28T14:22:48.585+0000 [conn23603] end connection 10.128.229.18:56771 (28 connections now open)
2014-11-28T14:22:48.585+0000 [conn23613] end connection 10.128.229.18:56792 (28 connections now open)
2014-11-28T14:22:48.585+0000 [conn23582] end connection 10.128.229.18:56722 (28 connections now open)
2014-11-28T14:22:48.585+0000 [conn23599] end connection 10.128.229.18:56751 (28 connections now open)
2014-11-28T14:22:48.586+0000 [conn23600] end connection 10.128.229.18:56760 (28 connections now open)
2014-11-28T14:22:48.586+0000 [conn24574] end connection 10.128.229.18:57240 (28 connections now open)
2014-11-28T14:22:48.587+0000 [conn23621] end connection 10.128.229.18:56816 (27 connections now open)
2014-11-28T14:22:48.588+0000 [conn23638] end connection 10.128.229.18:56849 (26 connections now open)
2014-11-28T14:22:48.585+0000 [conn23605] end connection 10.128.229.18:56775 (28 connections now open)
2014-11-28T14:22:48.585+0000 [conn23615] end connection 10.128.229.18:56796 (28 connections now open)
2014-11-28T14:22:48.589+0000 [initandlisten] connection accepted from 127.0.0.1:48476 #24655 (23 connections now open)
2014-11-28T14:22:48.587+0000 [conn23580] end connection 10.128.229.18:56717 (27 connections now open)
2014-11-28T14:22:48.587+0000 [conn23617] end connection 10.128.229.18:56803 (27 connections now open)
2014-11-28T14:22:48.587+0000 [conn23645] end connection 10.128.229.18:56860 (27 connections now open)
2014-11-28T14:22:48.587+0000 [conn23586] end connection 10.128.229.18:56726 (27 connections now open)
2014-11-28T14:22:48.590+0000 [conn23579] end connection 10.128.229.18:56714 (17 connections now open)
2014-11-28T14:22:48.594+0000 [conn24539] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.128.229.15:59529] 
2014-11-28T14:22:48.590+0000 [conn24557] end connection w3.x.y.z:46654 (15 connections now open)
2014-11-28T14:22:48.594+0000 [conn24601] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [w2.x.y.z:48981] 
2014-11-28T14:22:48.595+0000 [conn24566] replSet PRIMARY
2014-11-28T14:22:48.596+0000 [conn24654]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.597+0000 [conn24655]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "uuuuuuu", key: "xxx" }
2014-11-28T14:22:48.598+0000 [conn24566] replSet replSetReconfig new config saved locally
2014-11-28T14:22:48.598+0000 [conn24566] command admin.$cmd command: replSetReconfig { replSetReconfig: { _id: "wsyrs", version: 8, members: [ { _id: 0, host: "rs0.wsy:27017" }, { _id: 1, host: "rs1.wsy:27017" }, { _id: 3, host: "arb.wsy:27017", arbiterOnly: true }, { _id: 4, host: "mongo-db-6m7q:27017", priority: 0.0 }, { _id: 5, host: "mongo-db-td6n:27017", priority: 0.0 } ] } } keyUpdates:0 numYields:0 locks(micros) W:6321 w:594 reslen:37 113ms
2014-11-28T14:22:48.599+0000 [conn24566] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:46724] 
2014-11-28T14:22:48.599+0000 [rsHealthPoll] replset info arb.wsy:27017 heartbeat failed, retrying
2014-11-28T14:22:48.600+0000 [initandlisten] connection accepted from 10.128.229.15:33397 #24656 (7 connections now open)
2014-11-28T14:22:48.599+0000 [rsHealthPoll] replSet member rs0.wsy:27017 is up
2014-11-28T14:22:48.601+0000 [rsHealthPoll] replSet member rs0.wsy:27017 is now in state SECONDARY
2014-11-28T14:22:48.601+0000 [rsMgr] can't see a majority of the set, relinquishing primary
2014-11-28T14:22:48.601+0000 [rsMgr] replSet relinquishing primary state
2014-11-28T14:22:48.601+0000 [rsMgr] replSet SECONDARY
2014-11-28T14:22:48.601+0000 [rsMgr] replSet closing client sockets after relinquishing primary
2014-11-28T14:22:48.602+0000 [conn24654] end connection 10.128.229.15:33396 (5 connections now open)
2014-11-28T14:22:48.602+0000 [conn24655] end connection 127.0.0.1:48476 (5 connections now open)
2014-11-28T14:22:48.603+0000 [conn24656] end connection 10.128.229.15:33397 (3 connections now open)
2014-11-28T14:22:48.607+0000 [rsHealthPoll] replSet member arb.wsy:27017 is up
2014-11-28T14:22:48.608+0000 [initandlisten] connection accepted from 10.128.229.15:33398 #24657 (4 connections now open)
2014-11-28T14:22:48.609+0000 [conn24657]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.611+0000 [initandlisten] connection accepted from 10.128.229.15:33399 #24658 (5 connections now open)
2014-11-28T14:22:48.612+0000 [conn24658]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.632+0000 [rsHealthPoll] replSet member mongo-db-td6n:27017 is up
2014-11-28T14:22:48.633+0000 [rsHealthPoll] replSet member mongo-db-td6n:27017 is now in state SECONDARY
2014-11-28T14:22:48.647+0000 [rsHealthPoll] replSet member mongo-db-6m7q:27017 is up
2014-11-28T14:22:48.647+0000 [rsHealthPoll] replSet member mongo-db-6m7q:27017 is now in state SECONDARY
2014-11-28T14:22:48.652+0000 [initandlisten] connection accepted from w3.x.y.z:48730 #24659 (6 connections now open)
2014-11-28T14:22:48.654+0000 [initandlisten] connection accepted from w3.x.y.z:48731 #24660 (7 connections now open)
2014-11-28T14:22:48.661+0000 [initandlisten] connection accepted from w2.x.y.z:50208 #24661 (8 connections now open)
2014-11-28T14:22:48.662+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:22:48.683+0000 [conn24659]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.685+0000 [conn24660]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.701+0000 [conn24661]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.703+0000 [rsHealthPoll] warning: Failed to connect to w.x.y.z:27017, reason: errno:111 Connection refused
2014-11-28T14:22:48.704+0000 [rsHealthPoll] couldn't connect to mongo-arb-6ovf:27017: couldn't connect to server mongo-arb-6ovf:27017 (w.x.y.z) failed, connection attempt failed
2014-11-28T14:22:48.806+0000 [conn24659] end connection w3.x.y.z:48730 (7 connections now open)
2014-11-28T14:22:48.818+0000 [initandlisten] connection accepted from w2.x.y.z:50209 #24662 (8 connections now open)
2014-11-28T14:22:48.851+0000 [initandlisten] connection accepted from w3.x.y.z:48732 #24663 (9 connections now open)
2014-11-28T14:22:48.856+0000 [conn24662]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.889+0000 [conn24663]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:48.972+0000 [conn24661] end connection w2.x.y.z:50208 (8 connections now open)
2014-11-28T14:22:49.026+0000 [conn24660] end connection w3.x.y.z:48731 (7 connections now open)
2014-11-28T14:22:50.032+0000 [initandlisten] connection accepted from 10.128.229.18:57380 #24664 (8 connections now open)
2014-11-28T14:22:50.034+0000 [conn24664]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:22:50.037+0000 [rsMgr] not electing self, mongo-db-td6n:27017 would veto with 'I don't think rs1.wsy:27017 is electable'
2014-11-28T14:22:50.120+0000 [conn24653] replSet info voting yea for rs0.wsy:27017 (0)
2014-11-28T14:22:50.608+0000 [rsHealthPoll] replSet member arb.wsy:27017 is now in state REMOVED
2014-11-28T14:22:51.083+0000 [rsMgr] replSet not trying to elect self as responded yea to someone else recently
2014-11-28T14:22:51.247+0000 [rsMgr] replSet not trying to elect self as responded yea to someone else recently
2014-11-28T14:22:51.854+0000 [rsMgr] replSet not trying to elect self as responded yea to someone else recently
2014-11-28T14:22:53.632+0000 [conn24657] end connection 10.128.229.15:33398 (7 connections now open)
2014-11-28T14:22:54.605+0000 [rsHealthPoll] replSet member rs0.wsy:27017 is now in state PRIMARY
2014-11-28T14:22:54.610+0000 [rsHealthPoll] replSet member arb.wsy:27017 is now in state ARBITER
2014-11-28T14:22:55.039+0000 [rsBackgroundSync] replSet syncing to: rs0.wsy:27017
2014-11-28T14:22:55.042+0000 [rsBackgroundSync] replset setting syncSourceFeedback to rs0.wsy:27017
2014-11-28T14:22:56.132+0000 [conn24662] end connection w2.x.y.z:50209 (6 connections now open)
2014-11-28T14:22:56.203+0000 [conn24663] end connection w3.x.y.z:48732 (5 connections now open)
2014-11-28T14:22:59.078+0000 [conn24653] end connection 10.128.229.15:33372 (4 connections now open)
2014-11-28T14:22:59.079+0000 [initandlisten] connection accepted from 10.128.229.15:33410 #24665 (5 connections now open)
2014-11-28T14:22:59.080+0000 [conn24665]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2014-11-28T14:23:01.906+0000 [initandlisten] connection accepted from 10.128.229.18:57385 #24666 (6 connections now open)
2014-11-28T14:23:02.315+0000 [conn24666] end connection 10.128.229.18:57385 (5 connections now open)
2014-11-28T14:23:03.138+0000 [conn24651] end connection w2.x.y.z:50150 (4 connections now open)
2014-11-28T14:23:03.175+0000 [initandlisten] connection accepted from w2.x.y.z:50216 #24667 (5 connections now open)
2014-11-28T14:23:03.215+0000 [conn24667]  authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }

Comment by Eric Milkie [ 19/Feb/14 ]

Looking at the log messages, it appears that one of your configured member's addresses is resolving to a mongos instance instead of mongod. Is that correct?

Comment by Nic Cottrell (Personal) [ 19/Feb/14 ]

Doing "conf = rs.conf();" then editing the conf struct to match and then running "rs.reconfig(conf);" works perfectly...

Comment by Nic Cottrell (Personal) [ 19/Feb/14 ]

Just had a very similar problem (now on 2.4.9), and got this in the logs:

Wed Feb 19 17:46:33.286 [conn325315] replSet replSetReconfig config object parses ok, 4 members specified
Wed Feb 19 17:46:33.324 [conn325315] warning: could't check isSelf (n2-rs2.mongo.sprawk.com:27017) nextSafe(): { $err: "can't use 'local' database through mongos", code: 13644 }
Wed Feb 19 17:46:33.328 [conn325315] warning: could't check isSelf (n2-rs2.mongo.sprawk.com:27017) nextSafe(): { $err: "can't use 'local' database through mongos", code: 13644 }
Wed Feb 19 17:46:33.333 [conn325315] warning: could't check isSelf (n2-rs2.mongo.sprawk.com:27017) nextSafe(): { $err: "can't use 'local' database through mongos", code: 13644 }
Wed Feb 19 17:46:33.336 [conn325315] replSet cmufcc requestHeartbeat n2-rs2.mongo.sprawk.com:27017 : 13106 nextSafe(): { $err: "can't use 'local' database through mongos", code: 13644 }
Wed Feb 19 17:46:33.336 [conn325315]   Assertion failure m->h().toString() == i->h.toString() src/mongo/db/repl/rs_initiate.cpp 125
0xde46e1 0xda56bd 0xc11fe2 0xbe46e5 0xbe4ee4 0x8e049a 0x8e0f1d 0x8e2472 0xa85630 0xa89efc 0x9fe119 0x9ff633 0x6e8518 0xdd0cae 0x7f0d3a1c89d1 0x7f0d3956db6d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde46e1]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda56bd]
 /usr/bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x2672) [0xc11fe2]
 /usr/bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x745) [0xbe46e5]
 /usr/bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x64) [0xbe4ee4]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0x8e049a]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x71d) [0x8e0f1d]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x5f2) [0x8e2472]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x40) [0xa85630]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xd7c) [0xa89efc]
 /usr/bin/mongod() [0x9fe119]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x383) [0x9ff633]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e8518]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdd0cae]
 /lib64/libpthread.so.0(+0x79d1) [0x7f0d3a1c89d1]
 /lib64/libc.so.6(clone+0x6d) [0x7f0d3956db6d]

I'm accessing the primary via SSH and then "mongo localhost/admin"

Comment by Daniel Pasette (Inactive) [ 02/Jul/13 ]

Glad to hear you're back up. Without the log files, it will be hard to track down the chain of events.

Comment by Nic Cottrell (Personal) [ 02/Jul/13 ]

I ended up hacking the /etc/hosts to point sprawk.member0.mongolayer.com to my replacement server and after 5 minutes, the primary was elected. I then could add the server with the new hostname, then removed this old hostname. Everything now works.

Comment by Daniel Pasette (Inactive) [ 01/Jul/13 ]

If you have the log file from the time period when you ran the reconfig command, it should be possible to say what happened.

Generated at Thu Feb 08 03:22:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.