Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Duplicate
Priority: Critical - P2
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None

Operating System:
ALL
CAR Domain/s:
None

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name(s):
None
Goal Link:
None

3 AWS micro instances.
2 nodes, 1 arb 2.2.2

./bin/mongod --dbpath ./data --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1

./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1 --smallfiles

./bin/mongod --dbpath ./data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles

mongo shell connected to ip-10-227-21-162

rs1:SECONDARY> rs.status()
{
	"set" : "rs1",
	"date" : ISODate("2013-02-27T17:30:21Z"),
	"myState" : 2,
	"syncingTo" : "ip-10-48-238-94:27017",
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-227-21-162:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 66,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"errmsg" : "syncing to: ip-10-48-238-94:27017",
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-238-94:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 33,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
			"pingMs" : 1
		},
		{
			"_id" : 2,
			"name" : "ip-10-48-238-94:30000",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 33,
			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}
rs1:SECONDARY>

upgrade node ip-10-48-238-94, use --sslCRLFile

./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslCAFile ../cacert.pem --replSet rs1 --smallfiles --sslCRLFile ../crl.pem

same shell shows status

rs1:PRIMARY> rs.status()
{
	"set" : "rs1",
	"date" : ISODate("2013-02-27T17:30:35Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-227-21-162:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 80,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-238-94:27017",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"lastHeartbeat" : ISODate("2013-02-27T17:30:22Z"),
			"pingMs" : 0,
			"errmsg" : "DBClientBase::findN: transport error: ip-10-48-238-94:27017 ns: admin.$cmd query: { replSetHeartbeat: \"rs1\", v: 3, pv: 1, checkEmpty: false, from: \"ip-10-227-21-162:27017\", $auth: {} }"
		},
		{
			"_id" : 2,
			"name" : "ip-10-48-238-94:30000",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 47,
			"lastHeartbeat" : ISODate("2013-02-27T17:30:34Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}

upgrade arbiter, ip-10-48-238-94

./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles --sslCAFile ../cacert.pem --sslCRLFile ../crl.pem

call rs.status() on same shell

rs1:PRIMARY> rs.status()
Wed Feb 27 17:31:28.726 DBClientCursor::init call() failed
Wed Feb 27 17:31:28.727 JavaScript execution failed: Error: error doing query: failed at src/mongo/shell/query.js:L78
Wed Feb 27 17:31:28.728 trying reconnect to ip-10-227-21-162:27017
Wed Feb 27 17:31:28.732 reconnect ip-10-227-21-162:27017 ok
rs1:SECONDARY>

SIGSEGV at ip-10-227-21-162 (node not yet upgraded)

Wed Feb 27 17:31:02 [rsMgr] replSet SECONDARY
Wed Feb 27 17:31:02 [rsMgr] replSet closing client sockets after relinquishing primary
Wed Feb 27 17:31:03 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:04 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.48.238.94:37838] 
Wed Feb 27 17:31:04 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:05 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:06 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:07 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:07 [rsMgr] replSet can't see a majority, will not try to elect self
Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [conn13] end connection 10.48.238.94:37845 (3 connections now open)
Wed Feb 27 17:31:13 [initandlisten] connection accepted from 10.48.238.94:37852 #16 (4 connections now open)
Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 Invalid access at address: 0xf8 from thread: conn6

Wed Feb 27 17:31:27 Got signal: 11 (Segmentation fault).

Wed Feb 27 17:31:27 [initandlisten] connection accepted from 10.226.133.10:32776 #17 (5 connections now open)
Wed Feb 27 17:31:28 Backtrace:
0xa2a6d3 0x580264 0x580658 0x7fd31e031cb0 0x7fd31ddf1e1c 0x7fd31ddf2c95 0x7fd31ddf031c 0xa16099 0xa1956b 0xa0e584 0xa115ff 0x7fd31e029e9a 0x7fd31c2e8cbd 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xa2a6d3]
 ./bin/mongod(_ZN5mongo10abruptQuitEi+0x3e4) [0x580264]
 ./bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x268) [0x580658]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fd31e031cb0]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_n+0x19c) [0x7fd31ddf1e1c]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_bytes+0x3f5) [0x7fd31ddf2c95]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read+0x5c) [0x7fd31ddf031c]
 ./bin/mongod(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0xa16099]
 ./bin/mongod(_ZN5mongo6Socket4recvEPci+0x2b) [0xa1956b]
 ./bin/mongod(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x84) [0xa0e584]
 ./bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x41f) [0xa115ff]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd31e029e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd31c2e8cbd]

Assignee:: Unassigned
Reporter:: Gregor Macadam (Inactive)
Participants:: Eric Milkie, Gregor Macadam
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Feb 27 2013 05:37:59 PM UTC
Updated:: Mar 08 2013 03:55:49 PM UTC
Resolved:: Feb 27 2013 05:58:17 PM UTC

Details

Description

Attachments

Activity

People

Dates