[SERVER-8841] Once added a non-SSL node to an SSL RS, unable to add/remove any other node / kill reconfig operation Created: 04/Mar/13  Updated: 08/Mar/13  Resolved: 04/Mar/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Gregor Macadam Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-8734 Add non ssl arbiter to ssl RS crashes... Closed
Operating System: ALL
Participants:

 Description   

Two node RS, both SSL, add a non-ssl node. Cannot then add any further nodes. Or remove existing node.



 Comments   
Comment by Gregor Macadam [ 04/Mar/13 ]

Actually just need to shut down non-ssl node - my bad.

Comment by Gregor Macadam [ 04/Mar/13 ]

PRIMARY

ubuntu@ip-10-48-238-94:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Mon Mar  4 17:14:15.828 [initandlisten] MongoDB starting : pid=3040 port=27017 dbpath=./data/ 64-bit host=ip-10-48-238-94
Mon Mar  4 17:14:15.828 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Mon Mar  4 17:14:15.828 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Mon Mar  4 17:14:15.828 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Mon Mar  4 17:14:15.828 [initandlisten] allocator: tcmalloc
Mon Mar  4 17:14:15.828 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Mon Mar  4 17:14:15.879 [initandlisten] journal dir=./data/journal
Mon Mar  4 17:14:15.879 [initandlisten] recover : no journal files present, no recovery needed
Mon Mar  4 17:14:16.667 [initandlisten] preallocateIsFaster=true 7.52
Mon Mar  4 17:14:17.451 [initandlisten] preallocateIsFaster=true 7.76
Mon Mar  4 17:14:19.211 [initandlisten] preallocateIsFaster=true 7.46
Mon Mar  4 17:14:19.211 [initandlisten] preallocateIsFaster check took 3.332 secs
Mon Mar  4 17:14:19.211 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Mon Mar  4 17:14:23.091 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Mon Mar  4 17:14:27.158 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Mon Mar  4 17:14:31.035 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Mon Mar  4 17:14:31.035 [FileAllocator] creating directory ./data/_tmp
Mon Mar  4 17:14:31.051 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Mon Mar  4 17:14:31.051 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Mon Mar  4 17:14:31.059 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Mon Mar  4 17:14:31.062 [initandlisten] waiting for connections on port 27017 ssl
Mon Mar  4 17:14:31.063 [websvr] admin web console waiting for connections on port 28017 ssl
Mon Mar  4 17:14:31.066 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:31.067 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Mon Mar  4 17:14:41.068 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:45.886 [initandlisten] connection accepted from 10.36.129.84:53877 #1 (1 connection now open)
Mon Mar  4 17:14:50.275 [conn1] replSet replSetInitiate admin command received from client
Mon Mar  4 17:14:50.277 [conn1] replSet info initiate : no configuration specified.  Using a default configuration for the set
Mon Mar  4 17:14:50.277 [conn1] replSet created this configuration for initiation : { _id: "rs2", members: [ { _id: 0, host: "ip-10-48-238-94:27017" } ] }
Mon Mar  4 17:14:50.277 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Mon Mar  4 17:14:50.279 [conn1] replSet replSetInitiate all members seem up
Mon Mar  4 17:14:50.279 [conn1] ******
Mon Mar  4 17:14:50.279 [conn1] creating replication oplog of size: 990MB...
Mon Mar  4 17:14:50.280 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Mon Mar  4 17:14:50.295 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.014 secs
Mon Mar  4 17:14:50.295 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Mon Mar  4 17:14:50.311 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.015 secs
Mon Mar  4 17:14:50.339 [conn1] ******
Mon Mar  4 17:14:50.339 [conn1] replSet info saving a newer config version to local.system.replset
Mon Mar  4 17:14:50.339 [conn1] replSet saveConfigLocally done
Mon Mar  4 17:14:50.339 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
Mon Mar  4 17:14:51.088 [rsStart] replSet I am ip-10-48-238-94:27017
Mon Mar  4 17:14:51.088 [rsStart] replSet STARTUP2
Mon Mar  4 17:14:52.097 [rsSync] replSet SECONDARY
Mon Mar  4 17:14:52.097 [rsMgr] replSet info electSelf 0
Mon Mar  4 17:14:53.090 [rsMgr] replSet PRIMARY
Mon Mar  4 17:15:15.356 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Mon Mar  4 17:15:15.400 [conn1] replSet replSetReconfig [2]
Mon Mar  4 17:15:15.401 [conn1] replSet info saving a newer config version to local.system.replset
Mon Mar  4 17:15:15.431 [conn1] replSet saveConfigLocally done
Mon Mar  4 17:15:15.431 [conn1] replSet info : additive change to configuration
Mon Mar  4 17:15:15.431 [conn1] replSet replSetReconfig new config saved locally
Mon Mar  4 17:15:15.432 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is up
Mon Mar  4 17:15:15.433 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Mon Mar  4 17:15:25.276 [initandlisten] connection accepted from 10.226.133.10:45580 #2 (2 connections now open)
Mon Mar  4 17:15:25.284 [conn2] end connection 10.226.133.10:45580 (1 connection now open)
Mon Mar  4 17:15:25.285 [initandlisten] connection accepted from 10.226.133.10:45581 #3 (2 connections now open)
Mon Mar  4 17:15:25.439 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state STARTUP2
Mon Mar  4 17:15:41.398 [initandlisten] connection accepted from 10.226.133.10:45582 #4 (3 connections now open)
Mon Mar  4 17:15:41.522 [conn4] end connection 10.226.133.10:45582 (2 connections now open)
Mon Mar  4 17:15:42.313 [initandlisten] connection accepted from 10.226.133.10:45583 #5 (3 connections now open)
Mon Mar  4 17:15:42.527 [initandlisten] connection accepted from 10.226.133.10:45584 #6 (4 connections now open)
Mon Mar  4 17:15:43.457 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state RECOVERING
Mon Mar  4 17:15:43.541 [slaveTracking] build index local.slaves { _id: 1 }
Mon Mar  4 17:15:43.542 [slaveTracking] build index done.  scanned 0 total records. 0.001 secs
Mon Mar  4 17:15:45.458 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state SECONDARY
Mon Mar  4 17:15:53.327 [conn3] end connection 10.226.133.10:45581 (3 connections now open)
Mon Mar  4 17:15:53.328 [initandlisten] connection accepted from 10.226.133.10:45585 #7 (4 connections now open)
Mon Mar  4 17:16:07.787 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
Mon Mar  4 17:16:23.356 [conn7] end connection 10.226.133.10:45585 (3 connections now open)
Mon Mar  4 17:16:23.357 [initandlisten] connection accepted from 10.226.133.10:45586 #8 (4 connections now open)
Mon Mar  4 17:16:28.075 [initandlisten] connection accepted from 10.36.129.84:53878 #9 (5 connections now open)
Mon Mar  4 17:16:29.569 [conn9] going to kill op: op: 87
Mon Mar  4 17:16:29.570 [conn9] end connection 10.36.129.84:53878 (4 connections now open)
Mon Mar  4 17:16:31.482 [initandlisten] connection accepted from 10.36.129.84:53879 #10 (5 connections now open)
Mon Mar  4 17:16:53.387 [conn8] end connection 10.226.133.10:45586 (4 connections now open)
Mon Mar  4 17:16:53.388 [initandlisten] connection accepted from 10.226.133.10:45587 #11 (5 connections now open)
Mon Mar  4 17:17:23.411 [conn11] end connection 10.226.133.10:45587 (4 connections now open)
Mon Mar  4 17:17:23.412 [initandlisten] connection accepted from 10.226.133.10:45588 #12 (5 connections now open)
Mon Mar  4 17:17:52.703 [conn10] going to kill op: op: 87.0
Mon Mar  4 17:17:53.438 [conn12] end connection 10.226.133.10:45588 (4 connections now open)
Mon Mar  4 17:17:53.440 [initandlisten] connection accepted from 10.226.133.10:45589 #13 (5 connections now open)
Mon Mar  4 17:18:07.241 [conn10] going to kill op: op: 87.0
Mon Mar  4 17:18:23.467 [conn13] end connection 10.226.133.10:45589 (4 connections now open)
Mon Mar  4 17:18:23.468 [initandlisten] connection accepted from 10.226.133.10:45590 #14 (5 connections now open)
Mon Mar  4 17:18:53.494 [conn14] end connection 10.226.133.10:45590 (4 connections now open)
Mon Mar  4 17:18:53.497 [initandlisten] connection accepted from 10.226.133.10:45591 #15 (5 connections now open)
Mon Mar  4 17:19:23.521 [conn15] end connection 10.226.133.10:45591 (4 connections now open)
Mon Mar  4 17:19:23.522 [initandlisten] connection accepted from 10.226.133.10:45592 #16 (5 connections now open)
Mon Mar  4 17:19:53.555 [conn16] end connection 10.226.133.10:45592 (4 connections now open)
Mon Mar  4 17:19:53.557 [initandlisten] connection accepted from 10.226.133.10:45593 #17 (5 connections now open)
Mon Mar  4 17:20:23.580 [conn17] end connection 10.226.133.10:45593 (4 connections now open)
Mon Mar  4 17:20:23.581 [initandlisten] connection accepted from 10.226.133.10:45594 #18 (5 connections now open)
Mon Mar  4 17:20:53.613 [conn18] end connection 10.226.133.10:45594 (4 connections now open)
Mon Mar  4 17:20:53.617 [initandlisten] connection accepted from 10.226.133.10:45595 #19 (5 connections now open)
Mon Mar  4 17:21:23.645 [conn19] end connection 10.226.133.10:45595 (4 connections now open)
Mon Mar  4 17:21:23.646 [initandlisten] connection accepted from 10.226.133.10:45596 #20 (5 connections now open)
Mon Mar  4 17:21:53.676 [conn20] end connection 10.226.133.10:45596 (4 connections now open)
Mon Mar  4 17:21:53.677 [initandlisten] connection accepted from 10.226.133.10:45597 #21 (5 connections now open)
Mon Mar  4 17:22:23.704 [conn21] end connection 10.226.133.10:45597 (4 connections now open)
Mon Mar  4 17:22:23.705 [initandlisten] connection accepted from 10.226.133.10:45598 #22 (5 connections now open)

SECONDARY

ubuntu@ip-10-226-133-10:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Mon Mar  4 17:14:00.701 [initandlisten] MongoDB starting : pid=16350 port=27017 dbpath=./data/ 64-bit host=ip-10-226-133-10
Mon Mar  4 17:14:00.702 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Mon Mar  4 17:14:00.702 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Mon Mar  4 17:14:00.702 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Mon Mar  4 17:14:00.702 [initandlisten] allocator: tcmalloc
Mon Mar  4 17:14:00.702 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Mon Mar  4 17:14:00.746 [initandlisten] journal dir=./data/journal
Mon Mar  4 17:14:00.746 [initandlisten] recover : no journal files present, no recovery needed
Mon Mar  4 17:14:01.501 [initandlisten] preallocateIsFaster=true 7.2
Mon Mar  4 17:14:02.281 [initandlisten] preallocateIsFaster=true 7.54
Mon Mar  4 17:14:04.069 [initandlisten] preallocateIsFaster=true 7.76
Mon Mar  4 17:14:04.069 [initandlisten] preallocateIsFaster check took 3.323 secs
Mon Mar  4 17:14:04.069 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Mon Mar  4 17:14:07.825 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Mon Mar  4 17:14:11.637 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Mon Mar  4 17:14:15.487 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Mon Mar  4 17:14:15.487 [FileAllocator] creating directory ./data/_tmp
Mon Mar  4 17:14:15.505 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.008 secs
Mon Mar  4 17:14:15.505 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Mon Mar  4 17:14:15.512 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Mon Mar  4 17:14:15.516 [initandlisten] waiting for connections on port 27017 ssl
Mon Mar  4 17:14:15.517 [websvr] admin web console waiting for connections on port 28017 ssl
Mon Mar  4 17:14:15.520 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:15.521 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Mon Mar  4 17:14:25.522 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:35.522 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:45.523 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:55.524 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:05.524 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:05.613 [initandlisten] connection accepted from 10.48.238.94:51246 #1 (1 connection now open)
Mon Mar  4 17:15:05.624 [conn1] end connection 10.48.238.94:51246 (0 connections now open)
Mon Mar  4 17:15:05.625 [initandlisten] connection accepted from 10.48.238.94:51247 #2 (2 connections now open)
Mon Mar  4 17:15:15.538 [rsStart] trying to contact ip-10-48-238-94:27017
Mon Mar  4 17:15:15.561 [rsStart] replSet I am ip-10-226-133-10:27017
Mon Mar  4 17:15:15.561 [rsStart] replSet got config version 2 from a remote, saving locally
Mon Mar  4 17:15:15.561 [rsStart] replSet info saving a newer config version to local.system.replset
Mon Mar  4 17:15:15.562 [rsStart] replSet saveConfigLocally done
Mon Mar  4 17:15:15.563 [rsStart] replSet STARTUP2
Mon Mar  4 17:15:15.564 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Mon Mar  4 17:15:15.565 [rsHealthPoll] replSet member ip-10-48-238-94:27017 is up
Mon Mar  4 17:15:15.565 [rsHealthPoll] replSet member ip-10-48-238-94:27017 is now in state PRIMARY
Mon Mar  4 17:15:15.577 [rsSync] ******
Mon Mar  4 17:15:15.577 [rsSync] creating replication oplog of size: 990MB...
Mon Mar  4 17:15:15.579 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Mon Mar  4 17:15:15.596 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.016 secs
Mon Mar  4 17:15:15.596 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Mon Mar  4 17:15:15.605 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.008 secs
Mon Mar  4 17:15:15.633 [rsSync] ******
Mon Mar  4 17:15:15.633 [rsSync] replSet initial sync pending
Mon Mar  4 17:15:15.633 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
Mon Mar  4 17:15:31.634 [rsSync] replSet initial sync pending
Mon Mar  4 17:15:31.634 [rsSync] replSet syncing to: ip-10-48-238-94:27017
Mon Mar  4 17:15:31.666 [rsSync] build index local.me { _id: 1 }
Mon Mar  4 17:15:31.669 [rsSync] build index done.  scanned 0 total records. 0.002 secs
Mon Mar  4 17:15:31.672 [rsSync] build index local.replset.minvalid { _id: 1 }
Mon Mar  4 17:15:31.673 [rsSync] build index done.  scanned 0 total records. 0.001 secs
Mon Mar  4 17:15:31.673 [rsSync] replSet initial sync drop all databases
Mon Mar  4 17:15:31.674 [rsSync] dropAllDatabasesExceptLocal 1
Mon Mar  4 17:15:31.674 [rsSync] replSet initial sync clone all databases
Mon Mar  4 17:15:31.675 [rsSync] replSet initial sync data copy, starting syncup
Mon Mar  4 17:15:31.675 [rsSync] oplog sync 1 of 3
Mon Mar  4 17:15:31.676 [rsSync] oplog sync 2 of 3
Mon Mar  4 17:15:31.677 [rsSync] replSet initial sync building indexes
Mon Mar  4 17:15:31.677 [rsSync] oplog sync 3 of 3
Mon Mar  4 17:15:31.677 [rsSync] replSet initial sync finishing up
Mon Mar  4 17:15:31.714 [rsSync] replSet set minValid=5134d6a3:1
Mon Mar  4 17:15:31.776 [rsSync] replSet RECOVERING
Mon Mar  4 17:15:31.776 [rsSync] replSet initial sync done
Mon Mar  4 17:15:32.565 [rsBackgroundSync] replSet syncing to: ip-10-48-238-94:27017
Mon Mar  4 17:15:32.779 [rsSyncNotifier] replset setting oplog notifier to ip-10-48-238-94:27017
Mon Mar  4 17:15:33.701 [conn2] end connection 10.48.238.94:51247 (0 connections now open)
Mon Mar  4 17:15:33.703 [initandlisten] connection accepted from 10.48.238.94:51248 #3 (1 connection now open)
Mon Mar  4 17:15:33.777 [rsSync] replSet SECONDARY
Mon Mar  4 17:16:03.727 [conn3] end connection 10.48.238.94:51248 (0 connections now open)
Mon Mar  4 17:16:03.728 [initandlisten] connection accepted from 10.48.238.94:51250 #4 (1 connection now open)
Mon Mar  4 17:16:33.755 [conn4] end connection 10.48.238.94:51250 (0 connections now open)
Mon Mar  4 17:16:33.756 [initandlisten] connection accepted from 10.48.238.94:51251 #5 (1 connection now open)
Mon Mar  4 17:17:03.780 [conn5] end connection 10.48.238.94:51251 (0 connections now open)
Mon Mar  4 17:17:03.782 [initandlisten] connection accepted from 10.48.238.94:51252 #6 (1 connection now open)
Mon Mar  4 17:17:33.804 [conn6] end connection 10.48.238.94:51252 (0 connections now open)
Mon Mar  4 17:17:33.805 [initandlisten] connection accepted from 10.48.238.94:51253 #7 (1 connection now open)
Mon Mar  4 17:18:03.830 [conn7] end connection 10.48.238.94:51253 (0 connections now open)
Mon Mar  4 17:18:03.831 [initandlisten] connection accepted from 10.48.238.94:51254 #8 (1 connection now open)
Mon Mar  4 17:18:33.853 [conn8] end connection 10.48.238.94:51254 (0 connections now open)
Mon Mar  4 17:18:33.855 [initandlisten] connection accepted from 10.48.238.94:51255 #9 (1 connection now open)
Mon Mar  4 17:19:03.876 [conn9] end connection 10.48.238.94:51255 (0 connections now open)
Mon Mar  4 17:19:03.877 [initandlisten] connection accepted from 10.48.238.94:51256 #10 (1 connection now open)
Mon Mar  4 17:19:33.897 [conn10] end connection 10.48.238.94:51256 (0 connections now open)
Mon Mar  4 17:19:33.898 [initandlisten] connection accepted from 10.48.238.94:51257 #11 (1 connection now open)
Mon Mar  4 17:20:03.935 [conn11] end connection 10.48.238.94:51257 (0 connections now open)
Mon Mar  4 17:20:03.936 [initandlisten] connection accepted from 10.48.238.94:51258 #12 (1 connection now open)
Mon Mar  4 17:20:33.969 [conn12] end connection 10.48.238.94:51258 (0 connections now open)
Mon Mar  4 17:20:33.970 [initandlisten] connection accepted from 10.48.238.94:51259 #13 (1 connection now open)
Mon Mar  4 17:21:04.009 [conn13] end connection 10.48.238.94:51259 (0 connections now open)
Mon Mar  4 17:21:04.010 [initandlisten] connection accepted from 10.48.238.94:51260 #14 (1 connection now open)
Mon Mar  4 17:21:34.034 [conn14] end connection 10.48.238.94:51260 (0 connections now open)
Mon Mar  4 17:21:34.036 [initandlisten] connection accepted from 10.48.238.94:51261 #15 (1 connection now open)
Mon Mar  4 17:22:04.058 [conn15] end connection 10.48.238.94:51261 (0 connections now open)
Mon Mar  4 17:22:04.059 [initandlisten] connection accepted from 10.48.238.94:51262 #16 (1 connection now open)
Mon Mar  4 17:22:34.080 [conn16] end connection 10.48.238.94:51262 (0 connections now open)
Mon Mar  4 17:22:34.081 [initandlisten] connection accepted from 10.48.238.94:51263 #17 (1 connection now open)
Mon Mar  4 17:23:04.104 [conn17] end connection 10.48.238.94:51263 (0 connections now open)
Mon Mar  4 17:23:04.104 [initandlisten] connection accepted from 10.48.238.94:51264 #18 (1 connection now open)

ARBITER

ubuntu@ip-10-229-130-219:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/  --replSet rs2 --smallfiles 
Mon Mar  4 17:13:55.216 [initandlisten] MongoDB starting : pid=25529 port=27017 dbpath=./data/ 64-bit host=ip-10-229-130-219
Mon Mar  4 17:13:55.216 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Mon Mar  4 17:13:55.216 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Mon Mar  4 17:13:55.216 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Mon Mar  4 17:13:55.216 [initandlisten] allocator: tcmalloc
Mon Mar  4 17:13:55.216 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true }
Mon Mar  4 17:13:55.250 [initandlisten] journal dir=./data/journal
Mon Mar  4 17:13:55.250 [initandlisten] recover : no journal files present, no recovery needed
Mon Mar  4 17:13:55.972 [initandlisten] preallocateIsFaster=true 8.36
Mon Mar  4 17:13:56.718 [initandlisten] preallocateIsFaster=true 8.46
Mon Mar  4 17:13:58.397 [initandlisten] preallocateIsFaster=true 8.36
Mon Mar  4 17:13:58.397 [initandlisten] preallocateIsFaster check took 3.146 secs
Mon Mar  4 17:13:58.397 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Mon Mar  4 17:14:02.322 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Mon Mar  4 17:14:06.106 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Mon Mar  4 17:14:09.903 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Mon Mar  4 17:14:09.903 [FileAllocator] creating directory ./data/_tmp
Mon Mar  4 17:14:09.918 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Mon Mar  4 17:14:09.918 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Mon Mar  4 17:14:09.926 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Mon Mar  4 17:14:09.930 [initandlisten] waiting for connections on port 27017
Mon Mar  4 17:14:09.931 [websvr] admin web console waiting for connections on port 28017
Mon Mar  4 17:14:09.935 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:09.935 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Mon Mar  4 17:14:19.936 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:29.936 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:39.937 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:49.937 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:14:59.937 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:09.938 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:19.938 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:29.939 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:39.939 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:49.939 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:15:59.940 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:00.574 [initandlisten] connection accepted from 10.48.238.94:40752 #1 (1 connection now open)
Mon Mar  4 17:16:09.940 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:19.941 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:29.941 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:39.942 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:49.942 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:16:59.942 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:09.943 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:19.943 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:29.944 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:39.944 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:49.945 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:17:59.945 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:09.945 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:19.946 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:29.946 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:39.947 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:49.947 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:18:59.948 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:09.948 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:19.948 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:29.949 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:39.949 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:49.950 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:19:59.950 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:09.950 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:19.951 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:29.951 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:39.952 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:49.952 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:20:59.953 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:09.953 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:19.953 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:29.954 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:39.954 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:49.955 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:21:59.955 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:09.955 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:19.956 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:29.956 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:39.957 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:49.957 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:22:59.957 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:23:09.958 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:23:19.958 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:23:29.959 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Mon Mar  4 17:23:39.959 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

SHELL

ubuntu@ip-10-36-129-84:~/mongo$ ./mongo --ssl --sslPEMKeyFile ../gregorFreeBSD.pem --host ip-10-48-238-94
MongoDB shell version: 2.4.0-rc2-pre-
connecting to: ip-10-48-238-94:27017/test
Mon Mar  4 17:14:30.286 JavaScript execution failed: Error: couldn't connect to server ip-10-48-238-94:27017 at src/mongo/shell/mongo.js:L114
exception: connect failed
ubuntu@ip-10-36-129-84:~/mongo$ ./mongo --ssl --sslPEMKeyFile ../gregorFreeBSD.pem --host ip-10-48-238-94
MongoDB shell version: 2.4.0-rc2-pre-
connecting to: ip-10-48-238-94:27017/test
> rs.initiate()
{
	"info2" : "no configuration explicitly specified -- making one",
	"me" : "ip-10-48-238-94:27017",
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-04T17:15:00Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-48-238-94:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 45,
			"optime" : {
				"t" : 1362417290,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-04T17:14:50Z"),
			"self" : true
		}
	],
	"ok" : 1
}
rs2:PRIMARY> rs.add("ip-10-226-133-10")
{ "ok" : 1 }
rs2:PRIMARY> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-04T17:15:54Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-48-238-94:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 99,
			"optime" : {
				"t" : 1362417315,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-04T17:15:15Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-226-133-10:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 39,
			"optime" : {
				"t" : 1362417315,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-04T17:15:15Z"),
			"lastHeartbeat" : ISODate("2013-03-04T17:15:53Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : 1
		}
	],
	"ok" : 1
}
rs2:PRIMARY> rs.addArb("ip-10-229-130-219")
^C
do you want to kill the current op(s) on the server? (y/n): y
ubuntu@ip-10-36-129-84:~/mongo$ ./mongo --ssl --sslPEMKeyFile ../gregorFreeBSD.pem --host ip-10-48-238-94
MongoDB shell version: 2.4.0-rc2-pre-
connecting to: ip-10-48-238-94:27017/test
rs2:PRIMARY> rs.add("ip-10-36-129-84")
{ "ok" : 0, "errmsg" : "a replSetReconfig is already in progress" }
rs2:PRIMARY> db.currentOp()
{
	"inprog" : [
		{
			"opid" : 87,
			"active" : true,
			"secs_running" : 59,
			"op" : "query",
			"ns" : "",
			"query" : {
				"replSetReconfig" : {
					"_id" : "rs2",
					"version" : 3,
					"members" : [
						{
							"_id" : 0,
							"host" : "ip-10-48-238-94:27017"
						},
						{
							"_id" : 1,
							"host" : "ip-10-226-133-10:27017"
						},
						{
							"_id" : 2,
							"host" : "ip-10-229-130-219",
							"arbiterOnly" : true
						}
					]
				}
			},
			"client" : "10.36.129.84:53877",
			"desc" : "conn1",
			"threadId" : "0x7f54ec75f700",
			"connectionId" : 1,
			"waitingForLock" : false,
			"killPending" : true,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(1)
				},
				"timeAcquiringMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(2)
				}
			}
		},
		{
			"opid" : 139,
			"active" : true,
			"secs_running" : 4,
			"op" : "getmore",
			"ns" : "local.oplog.rs",
			"query" : {
				"ts" : {
					"$gte" : {
						"t" : 1362417315,
						"i" : 1
					}
				}
			},
			"client" : "10.226.133.10:45583",
			"desc" : "conn5",
			"threadId" : "0x7f5469721700",
			"connectionId" : 5,
			"waitingForLock" : false,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"r" : NumberLong(113),
					"w" : NumberLong(0)
				},
				"timeAcquiringMicros" : {
					"r" : NumberLong(21),
					"w" : NumberLong(0)
				}
			}
		}
	]
}
rs2:PRIMARY> db.killOp(87)
{ "info" : "attempting to kill op" }
rs2:PRIMARY> db.currentOp()
{
	"inprog" : [
		{
			"opid" : 87,
			"active" : true,
			"secs_running" : 106,
			"op" : "query",
			"ns" : "",
			"query" : {
				"replSetReconfig" : {
					"_id" : "rs2",
					"version" : 3,
					"members" : [
						{
							"_id" : 0,
							"host" : "ip-10-48-238-94:27017"
						},
						{
							"_id" : 1,
							"host" : "ip-10-226-133-10:27017"
						},
						{
							"_id" : 2,
							"host" : "ip-10-229-130-219",
							"arbiterOnly" : true
						}
					]
				}
			},
			"client" : "10.36.129.84:53877",
			"desc" : "conn1",
			"threadId" : "0x7f54ec75f700",
			"connectionId" : 1,
			"waitingForLock" : false,
			"killPending" : true,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(1)
				},
				"timeAcquiringMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(2)
				}
			}
		},
		{
			"opid" : 178,
			"active" : true,
			"secs_running" : 1,
			"op" : "getmore",
			"ns" : "local.oplog.rs",
			"query" : {
				"ts" : {
					"$gte" : {
						"t" : 1362417315,
						"i" : 1
					}
				}
			},
			"client" : "10.226.133.10:45583",
			"desc" : "conn5",
			"threadId" : "0x7f5469721700",
			"connectionId" : 5,
			"waitingForLock" : false,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"r" : NumberLong(37),
					"w" : NumberLong(0)
				},
				"timeAcquiringMicros" : {
					"r" : NumberLong(7),
					"w" : NumberLong(0)
				}
			}
		}
	]
}
rs2:PRIMARY> db.killOp(87)
{ "info" : "attempting to kill op" }
rs2:PRIMARY> db.currentOp()
{
	"inprog" : [
		{
			"opid" : 87,
			"active" : true,
			"secs_running" : 120,
			"op" : "query",
			"ns" : "",
			"query" : {
				"replSetReconfig" : {
					"_id" : "rs2",
					"version" : 3,
					"members" : [
						{
							"_id" : 0,
							"host" : "ip-10-48-238-94:27017"
						},
						{
							"_id" : 1,
							"host" : "ip-10-226-133-10:27017"
						},
						{
							"_id" : 2,
							"host" : "ip-10-229-130-219",
							"arbiterOnly" : true
						}
					]
				}
			},
			"client" : "10.36.129.84:53877",
			"desc" : "conn1",
			"threadId" : "0x7f54ec75f700",
			"connectionId" : 1,
			"waitingForLock" : false,
			"killPending" : true,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(1)
				},
				"timeAcquiringMicros" : {
					"R" : NumberLong(0),
					"W" : NumberLong(2)
				}
			}
		},
		{
			"opid" : 193,
			"active" : true,
			"secs_running" : 1,
			"op" : "getmore",
			"ns" : "local.oplog.rs",
			"query" : {
				"ts" : {
					"$gte" : {
						"t" : 1362417315,
						"i" : 1
					}
				}
			},
			"client" : "10.226.133.10:45583",
			"desc" : "conn5",
			"threadId" : "0x7f5469721700",
			"connectionId" : 5,
			"waitingForLock" : false,
			"numYields" : 0,
			"lockStats" : {
				"timeLockedMicros" : {
					"r" : NumberLong(56),
					"w" : NumberLong(0)
				},
				"timeAcquiringMicros" : {
					"r" : NumberLong(10),
					"w" : NumberLong(0)
				}
			}
		}
	]
}
rs2:PRIMARY> rs.add("ip-10-36-129-84")
{ "ok" : 0, "errmsg" : "a replSetReconfig is already in progress" }
rs2:PRIMARY> 
rs2:PRIMARY> rs.remove("ip-10-226-133-10:27017")
{ "ok" : 0, "errmsg" : "a replSetReconfig is already in progress" }
rs2:PRIMARY> 

Generated at Thu Feb 08 03:18:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.