Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8766

2 node 1 arb, 2.2.2 RS SSL, upgrade 2 nodes to 2.4.0-rc1, CRLFile, call rs.status() on shell connected to last 2.2.2 node, segfault

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL

      Description

      3 AWS micro instances.
      2 nodes, 1 arb 2.2.2

      ./bin/mongod --dbpath ./data --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1 

      ./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1 --smallfiles

      ./bin/mongod --dbpath ./data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles

      mongo shell connected to ip-10-227-21-162

      rs1:SECONDARY> rs.status()
      {
      	"set" : "rs1",
      	"date" : ISODate("2013-02-27T17:30:21Z"),
      	"myState" : 2,
      	"syncingTo" : "ip-10-48-238-94:27017",
      	"members" : [
      		{
      			"_id" : 0,
      			"name" : "ip-10-227-21-162:27017",
      			"health" : 1,
      			"state" : 2,
      			"stateStr" : "SECONDARY",
      			"uptime" : 66,
      			"optime" : {
      				"t" : 1361966564000,
      				"i" : 1
      			},
      			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
      			"errmsg" : "syncing to: ip-10-48-238-94:27017",
      			"self" : true
      		},
      		{
      			"_id" : 1,
      			"name" : "ip-10-48-238-94:27017",
      			"health" : 1,
      			"state" : 1,
      			"stateStr" : "PRIMARY",
      			"uptime" : 33,
      			"optime" : {
      				"t" : 1361966564000,
      				"i" : 1
      			},
      			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
      			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
      			"pingMs" : 1
      		},
      		{
      			"_id" : 2,
      			"name" : "ip-10-48-238-94:30000",
      			"health" : 1,
      			"state" : 7,
      			"stateStr" : "ARBITER",
      			"uptime" : 33,
      			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
      			"pingMs" : 0
      		}
      	],
      	"ok" : 1
      }
      rs1:SECONDARY> 

      upgrade node ip-10-48-238-94, use --sslCRLFile

      ./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslCAFile ../cacert.pem --replSet rs1 --smallfiles --sslCRLFile ../crl.pem 

      same shell shows status

      rs1:PRIMARY> rs.status()
      {
      	"set" : "rs1",
      	"date" : ISODate("2013-02-27T17:30:35Z"),
      	"myState" : 1,
      	"members" : [
      		{
      			"_id" : 0,
      			"name" : "ip-10-227-21-162:27017",
      			"health" : 1,
      			"state" : 1,
      			"stateStr" : "PRIMARY",
      			"uptime" : 80,
      			"optime" : {
      				"t" : 1361966564000,
      				"i" : 1
      			},
      			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
      			"self" : true
      		},
      		{
      			"_id" : 1,
      			"name" : "ip-10-48-238-94:27017",
      			"health" : 0,
      			"state" : 8,
      			"stateStr" : "(not reachable/healthy)",
      			"uptime" : 0,
      			"optime" : {
      				"t" : 1361966564000,
      				"i" : 1
      			},
      			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
      			"lastHeartbeat" : ISODate("2013-02-27T17:30:22Z"),
      			"pingMs" : 0,
      			"errmsg" : "DBClientBase::findN: transport error: ip-10-48-238-94:27017 ns: admin.$cmd query: { replSetHeartbeat: \"rs1\", v: 3, pv: 1, checkEmpty: false, from: \"ip-10-227-21-162:27017\", $auth: {} }"
      		},
      		{
      			"_id" : 2,
      			"name" : "ip-10-48-238-94:30000",
      			"health" : 1,
      			"state" : 7,
      			"stateStr" : "ARBITER",
      			"uptime" : 47,
      			"lastHeartbeat" : ISODate("2013-02-27T17:30:34Z"),
      			"pingMs" : 0
      		}
      	],
      	"ok" : 1
      }

      upgrade arbiter, ip-10-48-238-94

      ./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles --sslCAFile ../cacert.pem --sslCRLFile ../crl.pem 

      call rs.status() on same shell

      rs1:PRIMARY> rs.status()
      Wed Feb 27 17:31:28.726 DBClientCursor::init call() failed
      Wed Feb 27 17:31:28.727 JavaScript execution failed: Error: error doing query: failed at src/mongo/shell/query.js:L78
      Wed Feb 27 17:31:28.728 trying reconnect to ip-10-227-21-162:27017
      Wed Feb 27 17:31:28.732 reconnect ip-10-227-21-162:27017 ok
      rs1:SECONDARY> 

      SIGSEGV at ip-10-227-21-162 (node not yet upgraded)

      Wed Feb 27 17:31:02 [rsMgr] replSet SECONDARY
      Wed Feb 27 17:31:02 [rsMgr] replSet closing client sockets after relinquishing primary
      Wed Feb 27 17:31:03 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:04 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.48.238.94:37838] 
      Wed Feb 27 17:31:04 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:05 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:06 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:07 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:07 [rsMgr] replSet can't see a majority, will not try to elect self
      Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:13 [conn13] end connection 10.48.238.94:37845 (3 connections now open)
      Wed Feb 27 17:31:13 [initandlisten] connection accepted from 10.48.238.94:37852 #16 (4 connections now open)
      Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
      Wed Feb 27 17:31:27 Invalid access at address: 0xf8 from thread: conn6
       
      Wed Feb 27 17:31:27 Got signal: 11 (Segmentation fault).
       
      Wed Feb 27 17:31:27 [initandlisten] connection accepted from 10.226.133.10:32776 #17 (5 connections now open)
      Wed Feb 27 17:31:28 Backtrace:
      0xa2a6d3 0x580264 0x580658 0x7fd31e031cb0 0x7fd31ddf1e1c 0x7fd31ddf2c95 0x7fd31ddf031c 0xa16099 0xa1956b 0xa0e584 0xa115ff 0x7fd31e029e9a 0x7fd31c2e8cbd 
       ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xa2a6d3]
       ./bin/mongod(_ZN5mongo10abruptQuitEi+0x3e4) [0x580264]
       ./bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x268) [0x580658]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fd31e031cb0]
       /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_n+0x19c) [0x7fd31ddf1e1c]
       /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_bytes+0x3f5) [0x7fd31ddf2c95]
       /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read+0x5c) [0x7fd31ddf031c]
       ./bin/mongod(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0xa16099]
       ./bin/mongod(_ZN5mongo6Socket4recvEPci+0x2b) [0xa1956b]
       ./bin/mongod(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x84) [0xa0e584]
       ./bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x41f) [0xa115ff]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd31e029e9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd31c2e8cbd]

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            gregor Gregor Macadam
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: