[SERVER-8734] Add non ssl arbiter to ssl RS crashes primary Created: 26/Feb/13  Updated: 11/Jul/16  Resolved: 04/Mar/13

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: None
Fix Version/s: 2.2.4, 2.4.0-rc2

Type: Bug Priority: Major - P3
Reporter: Gregor Macadam Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File cacert.pem     File gregorFreeBSD.pem    
Issue Links:
Depends
Duplicate
is duplicated by SERVER-8735 Starting non-ssl mongod in an RS wher... Closed
is duplicated by SERVER-8730 Start ssl mongos where config server ... Closed
is duplicated by SERVER-8733 SSL mongos, add non-ssl shard, no inf... Closed
Related
related to SERVER-8841 Once added a non-SSL node to an SSL R... Closed
Operating System: ALL
Participants:

 Description   

rs2:PRIMARY> rs.addArb("ip-10-48-130-227:30000")
Tue Feb 26 14:25:28.412 DBClientCursor::init call() failed
Tue Feb 26 14:25:28.413 trying reconnect to ip-10-48-130-227:27017
Tue Feb 26 14:25:28.414 reconnect ip-10-48-130-227:27017 failed couldn't connect to server ip-10-48-130-227:27017
Tue Feb 26 14:25:28.415 Error: socket exception [CONNECT_ERROR] for ip-10-48-130-227:27017 src/mongo/shell/query.js:78
> 

Tue Feb 26 14:23:26.776 [rsMgr] replSet PRIMARY
Tue Feb 26 14:23:28.654 [initandlisten] connection accepted from 10.226.183.211:55502 #7 (2 connections now open)
Tue Feb 26 14:23:29.654 [initandlisten] connection accepted from 10.226.183.211:55503 #8 (3 connections now open)
Tue Feb 26 14:23:30.688 [slaveTracking] build index local.slaves { _id: 1 }
Tue Feb 26 14:23:30.703 [slaveTracking] build index done.  scanned 0 total records. 0.015 secs
Tue Feb 26 14:23:45.677 [conn6] end connection 10.226.183.211:55501 (2 connections now open)
Tue Feb 26 14:23:45.678 [initandlisten] connection accepted from 10.226.183.211:55504 #9 (3 connections now open)
Tue Feb 26 14:24:15.079 [initandlisten] connection accepted from 10.36.133.56:46549 #10 (4 connections now open)
Tue Feb 26 14:24:15.706 [conn9] end connection 10.226.183.211:55504 (3 connections now open)
Tue Feb 26 14:24:15.708 [initandlisten] connection accepted from 10.226.183.211:55505 #11 (4 connections now open)
Tue Feb 26 14:24:38.293 [conn10] replSet replSetReconfig config object parses ok, 3 members specified
Tue Feb 26 14:24:45.734 [conn11] end connection 10.226.183.211:55505 (3 connections now open)
Tue Feb 26 14:24:45.736 [initandlisten] connection accepted from 10.226.183.211:55506 #12 (4 connections now open)
Tue Feb 26 14:25:08.294 [conn10]   Fatal Assertion 16697
0xd971d3 0xd3df93 0xd8403d 0xd85e64 0xd7e411 0x751ed5 0x752b45 0xbc98ea 0xbcc732 0xbd77c8 0xc0a2d2 0xbe756a 0xbe7e14 0x8a98af 0x8aa1d7 0x8ab6cf 0xae128a 0xae57ae 0xa6e018 0x711884 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xd971d3]
 ./bin/mongod(_ZN5mongo13fassertFailedEi+0x63) [0xd3df93]
 ./bin/mongod(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xd8403d]
 ./bin/mongod(_ZN5mongo10SSLManager7connectEi+0x24) [0xd85e64]
 ./bin/mongod(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xd7e411]
 ./bin/mongod(_ZN5mongo18DBClientConnection8_connectERSs+0x415) [0x751ed5]
 ./bin/mongod(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0xc5) [0x752b45]
 ./bin/mongod(_ZN5mongo10ScopedConn7connectEv+0xea) [0xbc98ea]
 ./bin/mongod(_ZN5mongo10ScopedConnC2ERKSs+0xf2) [0xbcc732]
 ./bin/mongod(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x468) [0xbd77c8]
 ./bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x9e2) [0xc0a2d2]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4fa) [0xbe756a]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x5e4) [0xbe7e14]
 ./bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x8a98af]
 ./bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x657) [0x8aa1d7]
 ./bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x52f) [0x8ab6cf]
 ./bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0xae128a]
 ./bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xdee) [0xae57ae]
 ./bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xa58) [0xa6e018]
 ./bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x94) [0x711884]
Tue Feb 26 14:25:08.325 [conn10] 
 
***aborting after fassert() failure
 
 
Tue Feb 26 14:25:08.326 Got signal: 6 (Aborted).
 
Tue Feb 26 14:25:08.332 Backtrace:
0xd971d3 0x6fbb0f 0x7f36fcd0d4a0 0x7f36fcd0d425 0x7f36fcd10b8b 0xd3dfce 0xd8403d 0xd85e64 0xd7e411 0x751ed5 0x752b45 0xbc98ea 0xbcc732 0xbd77c8 0xc0a2d2 0xbe756a 0xbe7e14 0x8a98af 0x8aa1d7 0x8ab6cf 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xd971d3]
 ./bin/mongod(_ZN5mongo10abruptQuitEi+0x39f) [0x6fbb0f]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f36fcd0d4a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f36fcd0d425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f36fcd10b8b]
 ./bin/mongod(_ZN5mongo13fassertFailedEi+0x9e) [0xd3dfce]
 ./bin/mongod(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xd8403d]
 ./bin/mongod(_ZN5mongo10SSLManager7connectEi+0x24) [0xd85e64]
 ./bin/mongod(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xd7e411]
 ./bin/mongod(_ZN5mongo18DBClientConnection8_connectERSs+0x415) [0x751ed5]
 ./bin/mongod(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0xc5) [0x752b45]
 ./bin/mongod(_ZN5mongo10ScopedConn7connectEv+0xea) [0xbc98ea]
 ./bin/mongod(_ZN5mongo10ScopedConnC2ERKSs+0xf2) [0xbcc732]
 ./bin/mongod(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x468) [0xbd77c8]
 ./bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x9e2) [0xc0a2d2]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4fa) [0xbe756a]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x5e4) [0xbe7e14]
 ./bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x8a98af]
 ./bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x657) [0x8aa1d7]
 ./bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x52f) [0x8ab6cf]
 
ubuntu@ip-10-48-130-227:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.4.0-rc0$ 



 Comments   
Comment by auto [ 07/Mar/13 ]

Author:

{u'date': u'2013-03-07T17:36:23Z', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-8734 use a recursive mutex to work around OpenSSL deadlock
Branch: v2.2
https://github.com/mongodb/mongo/commit/70e051c0480212f80582713da5f0736d12f51de8

Comment by Gregor Macadam [ 04/Mar/13 ]

I'm satisfied that this no longer crashes and we can keep this bug closed - whether the RS is still in a valid state at this point is unknown and would suggest further testing.

Comment by Eric Milkie [ 04/Mar/13 ]

The crash is fixed. The deadlock fix should be backported.

Comment by Gregor Macadam [ 04/Mar/13 ]

Is this fixed then?

Comment by Eric Milkie [ 04/Mar/13 ]

Sorry, in all the noise I lost track of the fact that you're not starting the arbiter candidate with SSL. So the behavior we're seeing is actually correct. It won't crash now and you get a proper error in the log (connect timed out).

Comment by Gregor Macadam [ 01/Mar/13 ]

PRIMARY

ubuntu@ip-10-226-133-10:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Fri Mar  1 19:01:31.716 [initandlisten] MongoDB starting : pid=11854 port=27017 dbpath=./data/ 64-bit host=ip-10-226-133-10
Fri Mar  1 19:01:31.717 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 19:01:31.717 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Fri Mar  1 19:01:31.717 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 19:01:31.717 [initandlisten] allocator: tcmalloc
Fri Mar  1 19:01:31.717 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Fri Mar  1 19:01:31.738 [initandlisten] journal dir=./data/journal
Fri Mar  1 19:01:31.738 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 19:01:32.553 [initandlisten] preallocateIsFaster=true 8.24
Fri Mar  1 19:01:33.317 [initandlisten] preallocateIsFaster=true 7.26
Fri Mar  1 19:01:35.117 [initandlisten] preallocateIsFaster=true 7.86
Fri Mar  1 19:01:35.117 [initandlisten] preallocateIsFaster check took 3.379 secs
Fri Mar  1 19:01:35.117 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 19:01:38.821 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 19:01:42.522 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 19:01:47.305 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 19:01:47.305 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 19:01:47.321 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Fri Mar  1 19:01:47.321 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 19:01:47.329 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Fri Mar  1 19:01:47.333 [initandlisten] waiting for connections on port 27017 ssl
Fri Mar  1 19:01:47.334 [websvr] admin web console waiting for connections on port 28017 ssl
Fri Mar  1 19:01:47.337 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:01:47.338 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 19:01:57.339 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:07.339 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:09.848 [initandlisten] connection accepted from 10.36.129.84:40464 #1 (1 connection now open)
Fri Mar  1 19:02:15.467 [conn1] replSet replSetInitiate admin command received from client
Fri Mar  1 19:02:15.468 [conn1] replSet info initiate : no configuration specified.  Using a default configuration for the set
Fri Mar  1 19:02:15.468 [conn1] replSet created this configuration for initiation : { _id: "rs2", members: [ { _id: 0, host: "ip-10-226-133-10:27017" } ] }
Fri Mar  1 19:02:15.468 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Fri Mar  1 19:02:15.470 [conn1] replSet replSetInitiate all members seem up
Fri Mar  1 19:02:15.470 [conn1] ******
Fri Mar  1 19:02:15.470 [conn1] creating replication oplog of size: 990MB...
Fri Mar  1 19:02:15.471 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Fri Mar  1 19:02:15.485 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.014 secs
Fri Mar  1 19:02:15.486 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 19:02:15.498 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.011 secs
Fri Mar  1 19:02:15.521 [conn1] ******
Fri Mar  1 19:02:15.522 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 19:02:15.522 [conn1] replSet saveConfigLocally done
Fri Mar  1 19:02:15.522 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
Fri Mar  1 19:02:17.340 [rsStart] replSet I am ip-10-226-133-10:27017
Fri Mar  1 19:02:17.340 [rsStart] replSet STARTUP2
Fri Mar  1 19:02:18.342 [rsSync] replSet SECONDARY
Fri Mar  1 19:02:18.342 [rsMgr] replSet info electSelf 0
Fri Mar  1 19:02:19.341 [rsMgr] replSet PRIMARY
Fri Mar  1 19:02:41.389 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Fri Mar  1 19:02:41.406 [conn1] replSet replSetReconfig [2]
Fri Mar  1 19:02:41.407 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 19:02:41.407 [conn1] replSet saveConfigLocally done
Fri Mar  1 19:02:41.407 [conn1] replSet info : additive change to configuration
Fri Mar  1 19:02:41.407 [conn1] replSet replSetReconfig new config saved locally
Fri Mar  1 19:02:41.408 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is up
Fri Mar  1 19:02:41.408 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Fri Mar  1 19:02:50.055 [initandlisten] connection accepted from 10.48.159.19:57072 #2 (2 connections now open)
Fri Mar  1 19:02:50.065 [conn2] end connection 10.48.159.19:57072 (1 connection now open)
Fri Mar  1 19:02:50.066 [initandlisten] connection accepted from 10.48.159.19:57073 #3 (2 connections now open)
Fri Mar  1 19:02:51.414 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state STARTUP2
Fri Mar  1 19:03:06.112 [initandlisten] connection accepted from 10.48.159.19:57074 #4 (3 connections now open)
Fri Mar  1 19:03:06.191 [conn4] end connection 10.48.159.19:57074 (2 connections now open)
Fri Mar  1 19:03:07.081 [initandlisten] connection accepted from 10.48.159.19:57075 #5 (3 connections now open)
Fri Mar  1 19:03:07.194 [initandlisten] connection accepted from 10.48.159.19:57076 #6 (4 connections now open)
Fri Mar  1 19:03:07.424 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state RECOVERING
Fri Mar  1 19:03:08.208 [slaveTracking] build index local.slaves { _id: 1 }
Fri Mar  1 19:03:08.209 [slaveTracking] build index done.  scanned 0 total records. 0.001 secs
Fri Mar  1 19:03:09.435 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state SECONDARY
Fri Mar  1 19:03:18.095 [conn3] end connection 10.48.159.19:57073 (3 connections now open)
Fri Mar  1 19:03:18.096 [initandlisten] connection accepted from 10.48.159.19:57077 #7 (4 connections now open)
Fri Mar  1 19:03:23.088 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
Fri Mar  1 19:03:48.122 [conn7] end connection 10.48.159.19:57077 (3 connections now open)
Fri Mar  1 19:03:48.124 [initandlisten] connection accepted from 10.48.159.19:57078 #8 (4 connections now open)
Fri Mar  1 19:03:53.093 [conn1] ERROR: SSL error: 2, possibly timed out during connect
Fri Mar  1 19:03:53.093 [conn1] replSet cmufcc requestHeartbeat ip-10-229-130-219:30000 : 9001 socket exception [6] 
Fri Mar  1 19:03:53.093 [conn1] replSet replSetReconfig [2]
Fri Mar  1 19:03:53.093 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 19:03:53.094 [conn1] replSet saveConfigLocally done
Fri Mar  1 19:03:53.094 [conn1] replSet info : additive change to configuration
Fri Mar  1 19:03:53.094 [conn1] replSet replSetReconfig new config saved locally
Fri Mar  1 19:03:53.094 [conn1] command admin.$cmd command: { replSetReconfig: { _id: "rs2", version: 3, members: [ { _id: 0, host: "ip-10-226-133-10:27017" }, { _id: 1, host: "ip-10-48-159-19:27017" }, { _id: 2.0, host: "ip-10-229-130-219:30000", arbiterOnly: true } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:200 reslen:79 30006ms
Fri Mar  1 19:04:18.146 [conn8] end connection 10.48.159.19:57078 (3 connections now open)
Fri Mar  1 19:04:18.146 [initandlisten] connection accepted from 10.48.159.19:57080 #9 (4 connections now open)
Fri Mar  1 19:04:23.096 [rsHealthPoll] ERROR: SSL error: 2, possibly timed out during connect
Fri Mar  1 19:04:23.096 [rsHealthPoll] replSet info ip-10-229-130-219:30000 is down (or slow to respond): socket exception [CONNECT_ERROR] for 
Fri Mar  1 19:04:23.096 [rsHealthPoll] replSet member ip-10-229-130-219:30000 is now in state DOWN
Fri Mar  1 19:04:48.168 [conn9] end connection 10.48.159.19:57080 (3 connections now open)
Fri Mar  1 19:04:48.169 [initandlisten] connection accepted from 10.48.159.19:57082 #10 (4 connections now open)
Fri Mar  1 19:04:55.096 [rsHealthPoll] ERROR: SSL error: 2, possibly timed out during connect
Fri Mar  1 19:05:18.200 [conn10] end connection 10.48.159.19:57082 (3 connections now open)
Fri Mar  1 19:05:18.201 [initandlisten] connection accepted from 10.48.159.19:57084 #11 (4 connections now open)
^CFri Mar  1 19:05:20.620 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 19:05:20.621 [interruptThread] now exiting
Fri Mar  1 19:05:20.621 dbexit: 
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 19:05:20.621 [interruptThread] closing listening socket: 9
Fri Mar  1 19:05:20.621 [interruptThread] closing listening socket: 10
Fri Mar  1 19:05:20.621 [interruptThread] closing listening socket: 11
Fri Mar  1 19:05:20.621 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: final commit...
Fri Mar  1 19:05:20.621 [interruptThread] shutdown: closing all files...
Fri Mar  1 19:05:20.622 [interruptThread] closeAllFiles() finished
Fri Mar  1 19:05:20.622 [interruptThread] journalCleanup...
Fri Mar  1 19:05:20.622 [interruptThread] removeJournalFiles
Fri Mar  1 19:05:20.669 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 19:05:20.669 dbexit: really exiting now
ubuntu@ip-10-226-133-10:~$ 

SECONDARY

ubuntu@ip-10-48-159-19:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Fri Mar  1 19:01:51.601 [initandlisten] MongoDB starting : pid=14658 port=27017 dbpath=./data/ 64-bit host=ip-10-48-159-19
Fri Mar  1 19:01:51.602 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 19:01:51.602 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Fri Mar  1 19:01:51.602 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 19:01:51.602 [initandlisten] allocator: tcmalloc
Fri Mar  1 19:01:51.602 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Fri Mar  1 19:01:51.663 [initandlisten] journal dir=./data/journal
Fri Mar  1 19:01:51.663 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 19:01:52.527 [initandlisten] preallocateIsFaster=true 7.12
Fri Mar  1 19:01:53.419 [initandlisten] preallocateIsFaster=true 7.68
Fri Mar  1 19:01:55.199 [initandlisten] preallocateIsFaster=true 9.08
Fri Mar  1 19:01:55.199 [initandlisten] preallocateIsFaster check took 3.536 secs
Fri Mar  1 19:01:55.200 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 19:01:59.247 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 19:02:03.639 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 19:02:07.671 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 19:02:07.671 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 19:02:07.683 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.003 secs
Fri Mar  1 19:02:07.683 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 19:02:07.695 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.011 secs
Fri Mar  1 19:02:07.698 [initandlisten] waiting for connections on port 27017 ssl
Fri Mar  1 19:02:07.699 [websvr] admin web console waiting for connections on port 28017 ssl
Fri Mar  1 19:02:07.702 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:07.703 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 19:02:17.704 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:27.704 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:37.704 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:47.705 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:49.044 [initandlisten] connection accepted from 10.226.133.10:51428 #1 (1 connection now open)
Fri Mar  1 19:02:49.053 [conn1] end connection 10.226.133.10:51428 (0 connections now open)
Fri Mar  1 19:02:49.054 [initandlisten] connection accepted from 10.226.133.10:51429 #2 (1 connection now open)
Fri Mar  1 19:02:57.718 [rsStart] trying to contact ip-10-226-133-10:27017
Fri Mar  1 19:02:57.729 [rsStart] replSet I am ip-10-48-159-19:27017
Fri Mar  1 19:02:57.729 [rsStart] replSet got config version 2 from a remote, saving locally
Fri Mar  1 19:02:57.729 [rsStart] replSet info saving a newer config version to local.system.replset
Fri Mar  1 19:02:57.730 [rsStart] replSet saveConfigLocally done
Fri Mar  1 19:02:57.731 [rsStart] replSet STARTUP2
Fri Mar  1 19:02:57.732 [rsSync] ******
Fri Mar  1 19:02:57.732 [rsSync] creating replication oplog of size: 990MB...
Fri Mar  1 19:02:57.732 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Fri Mar  1 19:02:57.733 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Fri Mar  1 19:02:57.733 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is up
Fri Mar  1 19:02:57.733 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state PRIMARY
Fri Mar  1 19:02:57.755 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.022 secs
Fri Mar  1 19:02:57.755 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 19:02:57.763 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.007 secs
Fri Mar  1 19:02:57.763 [rsSync] ******
Fri Mar  1 19:02:57.764 [rsSync] replSet initial sync pending
Fri Mar  1 19:02:57.764 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
Fri Mar  1 19:03:13.764 [rsSync] replSet initial sync pending
Fri Mar  1 19:03:13.764 [rsSync] replSet syncing to: ip-10-226-133-10:27017
Fri Mar  1 19:03:13.781 [rsSync] build index local.me { _id: 1 }
Fri Mar  1 19:03:13.782 [rsSync] build index done.  scanned 0 total records. 0.001 secs
Fri Mar  1 19:03:13.789 [rsSync] build index local.replset.minvalid { _id: 1 }
Fri Mar  1 19:03:13.789 [rsSync] build index done.  scanned 0 total records. 0 secs
Fri Mar  1 19:03:13.789 [rsSync] replSet initial sync drop all databases
Fri Mar  1 19:03:13.789 [rsSync] dropAllDatabasesExceptLocal 1
Fri Mar  1 19:03:13.790 [rsSync] replSet initial sync clone all databases
Fri Mar  1 19:03:13.790 [rsSync] replSet initial sync data copy, starting syncup
Fri Mar  1 19:03:13.791 [rsSync] oplog sync 1 of 3
Fri Mar  1 19:03:13.791 [rsSync] oplog sync 2 of 3
Fri Mar  1 19:03:13.792 [rsSync] replSet initial sync building indexes
Fri Mar  1 19:03:13.792 [rsSync] oplog sync 3 of 3
Fri Mar  1 19:03:13.792 [rsSync] replSet initial sync finishing up
Fri Mar  1 19:03:13.844 [rsSync] replSet set minValid=5130fb51:1
Fri Mar  1 19:03:13.844 [rsSync] replSet RECOVERING
Fri Mar  1 19:03:13.844 [rsSync] replSet initial sync done
Fri Mar  1 19:03:14.733 [rsBackgroundSync] replSet syncing to: ip-10-226-133-10:27017
Fri Mar  1 19:03:14.846 [rsSyncNotifier] replset setting oplog notifier to ip-10-226-133-10:27017
Fri Mar  1 19:03:15.845 [rsSync] replSet SECONDARY
Fri Mar  1 19:03:17.079 [conn2] end connection 10.226.133.10:51429 (0 connections now open)
Fri Mar  1 19:03:17.080 [initandlisten] connection accepted from 10.226.133.10:51430 #3 (1 connection now open)
Fri Mar  1 19:03:47.110 [conn3] end connection 10.226.133.10:51430 (0 connections now open)
Fri Mar  1 19:03:47.112 [initandlisten] connection accepted from 10.226.133.10:51432 #4 (1 connection now open)
Fri Mar  1 19:04:01.790 [rsMgr] replset msgReceivedNewConfig version: version: 3
Fri Mar  1 19:04:01.790 [rsMgr] replSet info saving a newer config version to local.system.replset
Fri Mar  1 19:04:01.835 [rsMgr] replSet saveConfigLocally done
Fri Mar  1 19:04:01.835 [rsMgr] replSet info : additive change to configuration
Fri Mar  1 19:04:01.835 [rsMgr] replSet replSetReconfig new config saved locally
Fri Mar  1 19:04:17.154 [conn4] end connection 10.226.133.10:51432 (0 connections now open)
Fri Mar  1 19:04:17.155 [initandlisten] connection accepted from 10.226.133.10:51434 #5 (1 connection now open)
Fri Mar  1 19:04:31.839 [rsHealthPoll] ERROR: SSL error: 2, possibly timed out during connect
Fri Mar  1 19:04:31.839 [rsHealthPoll] replSet info ip-10-229-130-219:30000 is down (or slow to respond): socket exception [CONNECT_ERROR] for 
Fri Mar  1 19:04:31.839 [rsHealthPoll] replSet member ip-10-229-130-219:30000 is now in state DOWN
Fri Mar  1 19:04:47.184 [conn5] end connection 10.226.133.10:51434 (0 connections now open)
Fri Mar  1 19:04:47.185 [initandlisten] connection accepted from 10.226.133.10:51436 #6 (1 connection now open)
Fri Mar  1 19:05:03.839 [rsHealthPoll] ERROR: SSL error: 2, possibly timed out during connect
Fri Mar  1 19:05:17.210 [conn6] end connection 10.226.133.10:51436 (0 connections now open)
Fri Mar  1 19:05:17.212 [initandlisten] connection accepted from 10.226.133.10:51438 #7 (1 connection now open)
Fri Mar  1 19:05:28.282 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-10-226-133-10:27017
Fri Mar  1 19:05:28.283 [conn7] end connection 10.226.133.10:51438 (0 connections now open)
Fri Mar  1 19:05:29.870 [rsHealthPoll] DBClientCursor::init call() failed
Fri Mar  1 19:05:29.871 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 19:05:29.873 [rsHealthPoll] replSet info ip-10-226-133-10:27017 is down (or slow to respond): 
Fri Mar  1 19:05:29.873 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state DOWN
Fri Mar  1 19:05:29.873 [rsMgr] replSet can't see a majority, will not try to elect self
^CFri Mar  1 19:05:31.664 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 19:05:31.664 [interruptThread] now exiting
Fri Mar  1 19:05:31.664 dbexit: 
Fri Mar  1 19:05:31.664 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 19:05:31.664 [interruptThread] closing listening socket: 9
Fri Mar  1 19:05:31.665 [interruptThread] closing listening socket: 10
Fri Mar  1 19:05:31.665 [interruptThread] closing listening socket: 11
Fri Mar  1 19:05:31.665 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: final commit...
Fri Mar  1 19:05:31.665 [interruptThread] shutdown: closing all files...
Fri Mar  1 19:05:31.665 [interruptThread] closeAllFiles() finished
Fri Mar  1 19:05:31.665 [interruptThread] journalCleanup...
Fri Mar  1 19:05:31.665 [interruptThread] removeJournalFiles
Fri Mar  1 19:05:31.707 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 19:05:31.707 dbexit: really exiting now
ubuntu@ip-10-48-159-19:~$ 

ARBITER

ubuntu@ip-10-229-130-219:~$ ./mongod_f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d --dbpath ./data/ --port 30000 --replSet rs2 --smallfiles
Fri Mar  1 19:02:02.833 [initandlisten] MongoDB starting : pid=9352 port=30000 dbpath=./data/ 64-bit host=ip-10-229-130-219
Fri Mar  1 19:02:02.833 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 19:02:02.834 [initandlisten] git version: f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d
Fri Mar  1 19:02:02.834 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 19:02:02.834 [initandlisten] allocator: tcmalloc
Fri Mar  1 19:02:02.834 [initandlisten] options: { dbpath: "./data/", port: 30000, replSet: "rs2", smallfiles: true }
Fri Mar  1 19:02:02.858 [initandlisten] journal dir=./data/journal
Fri Mar  1 19:02:02.859 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 19:02:02.881 [initandlisten] waiting for connections on port 30000
Fri Mar  1 19:02:02.881 [websvr] admin web console waiting for connections on port 31000
Fri Mar  1 19:02:02.884 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:02.885 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 19:02:12.886 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:22.886 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:32.886 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:42.887 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:02:52.887 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:02.888 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:12.888 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:22.888 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:26.384 [initandlisten] connection accepted from 10.226.133.10:50490 #1 (1 connection now open)
Fri Mar  1 19:03:32.889 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:42.889 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:52.890 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:03:56.389 [initandlisten] connection accepted from 10.226.133.10:50492 #2 (2 connections now open)
Fri Mar  1 19:03:56.425 [conn1] end connection 10.226.133.10:50490 (1 connection now open)
Fri Mar  1 19:03:57.477 [initandlisten] connection accepted from 10.48.159.19:45208 #3 (2 connections now open)
Fri Mar  1 19:04:02.890 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:04:12.890 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:04:22.891 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:04:28.389 [conn2] end connection 10.226.133.10:50492 (1 connection now open)
Fri Mar  1 19:04:28.390 [initandlisten] connection accepted from 10.226.133.10:50494 #4 (2 connections now open)
Fri Mar  1 19:04:29.474 [conn3] end connection 10.48.159.19:45208 (1 connection now open)
Fri Mar  1 19:04:29.475 [initandlisten] connection accepted from 10.48.159.19:45210 #5 (2 connections now open)
Fri Mar  1 19:04:32.891 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:04:42.892 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:04:52.892 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:05:00.388 [conn4] end connection 10.226.133.10:50494 (1 connection now open)
Fri Mar  1 19:05:00.390 [initandlisten] connection accepted from 10.226.133.10:50496 #6 (2 connections now open)
Fri Mar  1 19:05:01.472 [conn5] end connection 10.48.159.19:45210 (1 connection now open)
Fri Mar  1 19:05:01.474 [initandlisten] connection accepted from 10.48.159.19:45212 #7 (2 connections now open)
Fri Mar  1 19:05:02.893 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:05:12.893 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:05:22.894 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 19:05:23.961 [conn6] end connection 10.226.133.10:50496 (1 connection now open)
Fri Mar  1 19:05:27.339 [conn7] end connection 10.48.159.19:45212 (0 connections now open)
^CFri Mar  1 19:05:29.312 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 19:05:29.312 [interruptThread] now exiting
Fri Mar  1 19:05:29.312 dbexit: 
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 19:05:29.313 [interruptThread] closing listening socket: 9
Fri Mar  1 19:05:29.313 [interruptThread] closing listening socket: 10
Fri Mar  1 19:05:29.313 [interruptThread] closing listening socket: 11
Fri Mar  1 19:05:29.313 [interruptThread] removing socket file: /tmp/mongodb-30000.sock
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: final commit...
Fri Mar  1 19:05:29.313 [interruptThread] shutdown: closing all files...
Fri Mar  1 19:05:29.314 [interruptThread] closeAllFiles() finished
Fri Mar  1 19:05:29.314 [interruptThread] journalCleanup...
Fri Mar  1 19:05:29.314 [interruptThread] removeJournalFiles
Fri Mar  1 19:05:29.342 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 19:05:29.342 dbexit: really exiting now
ubuntu@ip-10-229-130-219:~$ 

SHELL

ubuntu@ip-10-36-129-84:~/mongo$ ./mongo --ssl --host ip-10-226-133-10 --sslPEMKeyFile ../gregorFreeBSD.pem 
MongoDB shell version: 2.4.0-rc2-pre-
connecting to: ip-10-226-133-10:27017/test
> rs.initiate()
{
	"info2" : "no configuration explicitly specified -- making one",
	"me" : "ip-10-226-133-10:27017",
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-01T19:02:40Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-226-133-10:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 69,
			"optime" : {
				"t" : 1362164535,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T19:02:15Z"),
			"self" : true
		}
	],
	"ok" : 1
}
rs2:PRIMARY> rs.add("ip-10-48-159-19:27017")
{ "ok" : 1 }
rs2:PRIMARY> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-01T19:03:11Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-226-133-10:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 100,
			"optime" : {
				"t" : 1362164561,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T19:02:41Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-159-19:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 30,
			"optime" : {
				"t" : 1362164561,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T19:02:41Z"),
			"lastHeartbeat" : ISODate("2013-03-01T19:03:09Z"),
			"lastHeartbeatRecv" : ISODate("2013-03-01T19:03:10Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}
rs2:PRIMARY> rs.addArb("ip-10-229-130-219:30000")
{ "down" : [ "ip-10-229-130-219:30000" ], "ok" : 1 }
rs2:PRIMARY> 

Comment by auto [ 01/Mar/13 ]

Author:

{u'date': u'2013-03-01T18:31:39Z', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-8734 do not crash on SSL connection failure
Branch: master
https://github.com/mongodb/mongo/commit/f5a99e393fca6ca8167b0e37b4a7d05e7d43ce1d

Comment by Gregor Macadam [ 01/Mar/13 ]

OK should have full logs this time.

PRIMARY

ubuntu@ip-10-226-133-10:~$ ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3 --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Fri Mar  1 18:08:56.897 [initandlisten] MongoDB starting : pid=11250 port=27017 dbpath=./data/ 64-bit host=ip-10-226-133-10
Fri Mar  1 18:08:56.898 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 18:08:56.898 [initandlisten] git version: 1f7902206e31305e336c155166a6c12ae0f72ab3
Fri Mar  1 18:08:56.898 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 18:08:56.898 [initandlisten] allocator: tcmalloc
Fri Mar  1 18:08:56.898 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Fri Mar  1 18:08:56.949 [initandlisten] journal dir=./data/journal
Fri Mar  1 18:08:56.949 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 18:08:57.825 [initandlisten] preallocateIsFaster=true 8.8
Fri Mar  1 18:08:58.697 [initandlisten] preallocateIsFaster=true 8.34
Fri Mar  1 18:09:00.553 [initandlisten] preallocateIsFaster=true 8.14
Fri Mar  1 18:09:00.554 [initandlisten] preallocateIsFaster check took 3.604 secs
Fri Mar  1 18:09:00.554 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 18:09:04.553 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 18:09:08.529 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 18:09:12.485 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 18:09:12.485 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 18:09:12.501 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Fri Mar  1 18:09:12.501 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 18:09:12.509 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Fri Mar  1 18:09:12.513 [initandlisten] waiting for connections on port 27017 ssl
Fri Mar  1 18:09:12.514 [websvr] admin web console waiting for connections on port 28017 ssl
Fri Mar  1 18:09:12.518 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:12.518 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 18:09:21.009 [initandlisten] connection accepted from 10.36.129.84:40455 #1 (1 connection now open)
Fri Mar  1 18:09:22.519 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:24.279 [conn1] replSet replSetInitiate admin command received from client
Fri Mar  1 18:09:24.281 [conn1] replSet info initiate : no configuration specified.  Using a default configuration for the set
Fri Mar  1 18:09:24.281 [conn1] replSet created this configuration for initiation : { _id: "rs2", members: [ { _id: 0, host: "ip-10-226-133-10:27017" } ] }
Fri Mar  1 18:09:24.281 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Fri Mar  1 18:09:24.284 [conn1] replSet replSetInitiate all members seem up
Fri Mar  1 18:09:24.285 [conn1] ******
Fri Mar  1 18:09:24.285 [conn1] creating replication oplog of size: 990MB...
Fri Mar  1 18:09:24.286 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Fri Mar  1 18:09:24.297 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.011 secs
Fri Mar  1 18:09:24.297 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 18:09:24.313 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.015 secs
Fri Mar  1 18:09:24.361 [conn1] ******
Fri Mar  1 18:09:24.361 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 18:09:24.421 [conn1] replSet saveConfigLocally done
Fri Mar  1 18:09:24.421 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
Fri Mar  1 18:09:24.421 [conn1] command admin.$cmd command: { replSetInitiate: undefined } ntoreturn:1 keyUpdates:0 locks(micros) W:137514 reslen:206 141ms
Fri Mar  1 18:09:32.520 [rsStart] replSet I am ip-10-226-133-10:27017
Fri Mar  1 18:09:32.520 [rsStart] replSet STARTUP2
Fri Mar  1 18:09:33.522 [rsSync] replSet SECONDARY
Fri Mar  1 18:09:33.522 [rsMgr] replSet info electSelf 0
Fri Mar  1 18:09:34.521 [rsMgr] replSet PRIMARY
Fri Mar  1 18:12:05.878 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Fri Mar  1 18:12:05.903 [conn1] replSet replSetReconfig [2]
Fri Mar  1 18:12:05.903 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 18:12:05.903 [conn1] replSet saveConfigLocally done
Fri Mar  1 18:12:05.904 [conn1] replSet info : additive change to configuration
Fri Mar  1 18:12:05.904 [conn1] replSet replSetReconfig new config saved locally
Fri Mar  1 18:12:05.904 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is up
Fri Mar  1 18:12:05.905 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Fri Mar  1 18:12:13.697 [initandlisten] connection accepted from 10.48.159.19:57038 #2 (2 connections now open)
Fri Mar  1 18:12:13.712 [conn2] end connection 10.48.159.19:57038 (1 connection now open)
Fri Mar  1 18:12:13.713 [initandlisten] connection accepted from 10.48.159.19:57039 #3 (2 connections now open)
Fri Mar  1 18:12:13.909 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state STARTUP2
Fri Mar  1 18:12:29.752 [initandlisten] connection accepted from 10.48.159.19:57040 #4 (3 connections now open)
Fri Mar  1 18:12:29.826 [conn4] end connection 10.48.159.19:57040 (2 connections now open)
Fri Mar  1 18:12:29.921 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state RECOVERING
Fri Mar  1 18:12:30.736 [initandlisten] connection accepted from 10.48.159.19:57041 #5 (3 connections now open)
Fri Mar  1 18:12:30.832 [initandlisten] connection accepted from 10.48.159.19:57042 #6 (4 connections now open)
Fri Mar  1 18:12:31.856 [slaveTracking] build index local.slaves { _id: 1 }
Fri Mar  1 18:12:31.858 [slaveTracking] build index done.  scanned 0 total records. 0.001 secs
Fri Mar  1 18:12:31.927 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state SECONDARY
Fri Mar  1 18:12:35.347 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
Fri Mar  1 18:12:41.754 [conn3] end connection 10.48.159.19:57039 (3 connections now open)
Fri Mar  1 18:12:41.755 [initandlisten] connection accepted from 10.48.159.19:57043 #7 (4 connections now open)
Fri Mar  1 18:13:05.352 [conn1]   Fatal Assertion 16697
0xb856b3 0xb4925d 0xb75f8d 0xb781d4 0xb733a1 0x74af5d 0x74b2a8 0xa20c8a 0xa231af 0xa2cb1b 0xa526bd 0xa38814 0xa38ac5 0x855edf 0x856821 0x857967 0x96841a 0x96b3c1 0x91553c 0x71a2f9 
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo15printStackTraceERSo+0x23) [0xb856b3]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo13fassertFailedEi+0x4d) [0xb4925d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb75f8d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager7connectEi+0x24) [0xb781d4]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb733a1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74af5d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74b2a8]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa20c8a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa231af]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa2cb1b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa526bd]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa38814]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa38ac5]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x855edf]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x856821]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x857967]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0x96841a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xcf1) [0x96b3c1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xb4c) [0x91553c]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x89) [0x71a2f9]
Fri Mar  1 18:13:05.360 [conn1] 
 
***aborting after fassert() failure
 
 
Fri Mar  1 18:13:05.360 Got signal: 6 (Aborted).
 
Fri Mar  1 18:13:05.366 Backtrace:
0xb856b3 0x70921c 0x7f326a0f74a0 0x7f326a0f7425 0x7f326a0fab8b 0xb49298 0xb75f8d 0xb781d4 0xb733a1 0x74af5d 0x74b2a8 0xa20c8a 0xa231af 0xa2cb1b 0xa526bd 0xa38814 0xa38ac5 0x855edf 0x856821 0x857967 
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo15printStackTraceERSo+0x23) [0xb856b3]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10abruptQuitEi+0x3fc) [0x70921c]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f326a0f74a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f326a0f7425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f326a0fab8b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo13fassertFailedEi+0x88) [0xb49298]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb75f8d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager7connectEi+0x24) [0xb781d4]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb733a1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74af5d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74b2a8]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa20c8a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa231af]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa2cb1b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa526bd]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa38814]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa38ac5]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x855edf]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x856821]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x857967]

SECONDARY

ubuntu@ip-10-48-159-19:~$ ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3 --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Fri Mar  1 18:08:55.385 [initandlisten] MongoDB starting : pid=14058 port=27017 dbpath=./data/ 64-bit host=ip-10-48-159-19
Fri Mar  1 18:08:55.386 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 18:08:55.386 [initandlisten] git version: 1f7902206e31305e336c155166a6c12ae0f72ab3
Fri Mar  1 18:08:55.386 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 18:08:55.386 [initandlisten] allocator: tcmalloc
Fri Mar  1 18:08:55.386 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Fri Mar  1 18:08:55.420 [initandlisten] journal dir=./data/journal
Fri Mar  1 18:08:55.420 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 18:08:56.223 [initandlisten] preallocateIsFaster=true 8.76
Fri Mar  1 18:08:57.023 [initandlisten] preallocateIsFaster=true 7.58
Fri Mar  1 18:08:58.887 [initandlisten] preallocateIsFaster=true 8.06
Fri Mar  1 18:08:58.887 [initandlisten] preallocateIsFaster check took 3.466 secs
Fri Mar  1 18:08:58.887 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 18:09:03.223 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 18:09:07.083 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 18:09:11.175 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 18:09:11.175 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 18:09:11.191 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Fri Mar  1 18:09:11.191 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 18:09:11.203 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.011 secs
Fri Mar  1 18:09:11.206 [initandlisten] waiting for connections on port 27017 ssl
Fri Mar  1 18:09:11.207 [websvr] admin web console waiting for connections on port 28017 ssl
Fri Mar  1 18:09:11.210 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:11.211 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 18:09:21.211 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:31.212 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:41.212 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:51.213 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:01.213 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:11.214 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:21.214 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:31.214 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:41.215 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:51.215 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:01.216 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:11.216 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:21.216 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:31.217 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:41.217 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:51.217 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:01.218 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:11.218 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:13.407 [initandlisten] connection accepted from 10.226.133.10:51411 #1 (1 connection now open)
Fri Mar  1 18:12:13.418 [conn1] end connection 10.226.133.10:51411 (0 connections now open)
Fri Mar  1 18:12:13.418 [initandlisten] connection accepted from 10.226.133.10:51412 #2 (1 connection now open)
Fri Mar  1 18:12:21.238 [rsStart] trying to contact ip-10-226-133-10:27017
Fri Mar  1 18:12:21.252 [rsStart] replSet I am ip-10-48-159-19:27017
Fri Mar  1 18:12:21.252 [rsStart] replSet got config version 2 from a remote, saving locally
Fri Mar  1 18:12:21.252 [rsStart] replSet info saving a newer config version to local.system.replset
Fri Mar  1 18:12:21.253 [rsStart] replSet saveConfigLocally done
Fri Mar  1 18:12:21.254 [rsStart] replSet STARTUP2
Fri Mar  1 18:12:21.255 [rsSync] ******
Fri Mar  1 18:12:21.255 [rsSync] creating replication oplog of size: 990MB...
Fri Mar  1 18:12:21.255 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Fri Mar  1 18:12:21.256 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is up
Fri Mar  1 18:12:21.256 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state PRIMARY
Fri Mar  1 18:12:21.256 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Fri Mar  1 18:12:21.267 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.011 secs
Fri Mar  1 18:12:21.267 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 18:12:21.275 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.007 secs
Fri Mar  1 18:12:21.276 [rsSync] ******
Fri Mar  1 18:12:21.276 [rsSync] replSet initial sync pending
Fri Mar  1 18:12:21.276 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
Fri Mar  1 18:12:37.276 [rsSync] replSet initial sync pending
Fri Mar  1 18:12:37.276 [rsSync] replSet syncing to: ip-10-226-133-10:27017
Fri Mar  1 18:12:37.292 [rsSync] build index local.me { _id: 1 }
Fri Mar  1 18:12:37.293 [rsSync] build index done.  scanned 0 total records. 0.001 secs
Fri Mar  1 18:12:37.300 [rsSync] build index local.replset.minvalid { _id: 1 }
Fri Mar  1 18:12:37.301 [rsSync] build index done.  scanned 0 total records. 0 secs
Fri Mar  1 18:12:37.301 [rsSync] replSet initial sync drop all databases
Fri Mar  1 18:12:37.301 [rsSync] dropAllDatabasesExceptLocal 1
Fri Mar  1 18:12:37.301 [rsSync] replSet initial sync clone all databases
Fri Mar  1 18:12:37.302 [rsSync] replSet initial sync data copy, starting syncup
Fri Mar  1 18:12:37.302 [rsSync] oplog sync 1 of 3
Fri Mar  1 18:12:37.303 [rsSync] oplog sync 2 of 3
Fri Mar  1 18:12:37.304 [rsSync] replSet initial sync building indexes
Fri Mar  1 18:12:37.304 [rsSync] oplog sync 3 of 3
Fri Mar  1 18:12:37.304 [rsSync] replSet initial sync finishing up
Fri Mar  1 18:12:37.351 [rsSync] replSet set minValid=5130ef75:1
Fri Mar  1 18:12:37.352 [rsSync] replSet RECOVERING
Fri Mar  1 18:12:37.352 [rsSync] replSet initial sync done
Fri Mar  1 18:12:38.256 [rsBackgroundSync] replSet syncing to: ip-10-226-133-10:27017
Fri Mar  1 18:12:38.352 [rsSyncNotifier] replset setting oplog notifier to ip-10-226-133-10:27017
Fri Mar  1 18:12:39.376 [rsSync] replSet SECONDARY
Fri Mar  1 18:12:41.459 [conn2] end connection 10.226.133.10:51412 (0 connections now open)
Fri Mar  1 18:12:41.464 [initandlisten] connection accepted from 10.226.133.10:51413 #3 (1 connection now open)
Fri Mar  1 18:13:11.498 [conn3] end connection 10.226.133.10:51413 (0 connections now open)
Fri Mar  1 18:13:11.500 [initandlisten] connection accepted from 10.226.133.10:51415 #4 (1 connection now open)
Fri Mar  1 18:13:12.904 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-10-226-133-10:27017
Fri Mar  1 18:13:12.904 [conn4] end connection 10.226.133.10:51415 (0 connections now open)
Fri Mar  1 18:13:13.306 [rsHealthPoll] DBClientCursor::init call() failed
Fri Mar  1 18:13:13.306 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 18:13:13.307 [rsHealthPoll] replSet info ip-10-226-133-10:27017 is down (or slow to respond): 
Fri Mar  1 18:13:13.308 [rsHealthPoll] replSet member ip-10-226-133-10:27017 is now in state DOWN
Fri Mar  1 18:13:13.308 [rsMgr] replSet can't see a majority, will not try to elect self
Fri Mar  1 18:13:15.309 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 18:13:17.311 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
^CFri Mar  1 18:13:19.271 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 18:13:19.271 [interruptThread] now exiting
Fri Mar  1 18:13:19.271 dbexit: 
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 18:13:19.272 [interruptThread] closing listening socket: 9
Fri Mar  1 18:13:19.272 [interruptThread] closing listening socket: 10
Fri Mar  1 18:13:19.272 [interruptThread] closing listening socket: 11
Fri Mar  1 18:13:19.272 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: final commit...
Fri Mar  1 18:13:19.272 [interruptThread] shutdown: closing all files...
Fri Mar  1 18:13:19.272 [interruptThread] closeAllFiles() finished
Fri Mar  1 18:13:19.273 [interruptThread] journalCleanup...
Fri Mar  1 18:13:19.273 [interruptThread] removeJournalFiles
Fri Mar  1 18:13:19.312 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 18:13:19.312 dbexit: really exiting now

ARBITER

ubuntu@ip-10-229-130-219:~$ ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3 --dbpath ./data/ --port 30000 --replSet rs2 --smallfiles
Fri Mar  1 18:08:36.189 [initandlisten] MongoDB starting : pid=8762 port=30000 dbpath=./data/ 64-bit host=ip-10-229-130-219
Fri Mar  1 18:08:36.189 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 18:08:36.189 [initandlisten] git version: 1f7902206e31305e336c155166a6c12ae0f72ab3
Fri Mar  1 18:08:36.189 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 18:08:36.189 [initandlisten] allocator: tcmalloc
Fri Mar  1 18:08:36.190 [initandlisten] options: { dbpath: "./data/", port: 30000, replSet: "rs2", smallfiles: true }
Fri Mar  1 18:08:36.230 [initandlisten] journal dir=./data/journal
Fri Mar  1 18:08:36.231 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 18:08:36.942 [initandlisten] preallocateIsFaster=true 7.26
Fri Mar  1 18:08:37.654 [initandlisten] preallocateIsFaster=true 7.8
Fri Mar  1 18:08:39.414 [initandlisten] preallocateIsFaster=true 5.54
Fri Mar  1 18:08:39.414 [initandlisten] preallocateIsFaster check took 3.183 secs
Fri Mar  1 18:08:39.414 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 18:08:43.378 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 18:08:47.258 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 18:08:51.084 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 18:08:51.085 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 18:08:51.106 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Fri Mar  1 18:08:51.106 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 18:08:51.114 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Fri Mar  1 18:08:51.118 [initandlisten] waiting for connections on port 30000
Fri Mar  1 18:08:51.118 [websvr] admin web console waiting for connections on port 31000
Fri Mar  1 18:08:51.122 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:08:51.122 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 18:09:01.123 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:11.124 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:21.124 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:31.124 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:41.125 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:09:51.125 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:01.126 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:11.126 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:21.127 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:31.127 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:41.127 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:10:51.128 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:01.128 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:11.129 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:21.129 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:31.130 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:41.130 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:11:51.130 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:01.131 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:11.131 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:21.132 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:31.132 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:38.653 [initandlisten] connection accepted from 10.226.133.10:50473 #1 (1 connection now open)
Fri Mar  1 18:12:41.133 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:12:51.133 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:13:01.133 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:13:08.677 [conn1] end connection 10.226.133.10:50473 (0 connections now open)
Fri Mar  1 18:13:11.134 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
^CFri Mar  1 18:13:17.129 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 18:13:17.129 [interruptThread] now exiting
Fri Mar  1 18:13:17.129 dbexit: 
Fri Mar  1 18:13:17.129 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 18:13:17.129 [interruptThread] closing listening socket: 9
Fri Mar  1 18:13:17.130 [interruptThread] closing listening socket: 10
Fri Mar  1 18:13:17.130 [interruptThread] closing listening socket: 11
Fri Mar  1 18:13:17.130 [interruptThread] removing socket file: /tmp/mongodb-30000.sock
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: final commit...
Fri Mar  1 18:13:17.130 [interruptThread] shutdown: closing all files...
Fri Mar  1 18:13:17.131 [interruptThread] closeAllFiles() finished
Fri Mar  1 18:13:17.131 [interruptThread] journalCleanup...
Fri Mar  1 18:13:17.131 [interruptThread] removeJournalFiles
Fri Mar  1 18:13:17.161 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 18:13:17.162 dbexit: really exiting now
ubuntu@ip-10-229-130-219:~$ 

SHELL

ubuntu@ip-10-36-129-84:~/mongo$ ./mongo --ssl --host ip-10-226-133-10 --sslPEMKeyFile ../gregorFreeBSD.pem 
MongoDB shell version: 2.4.0-rc2-pre-
connecting to: ip-10-226-133-10:27017/test

> rs.initiate()
{
	"info2" : "no configuration explicitly specified -- making one",
	"me" : "ip-10-226-133-10:27017",
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}
> rs.add("ip-10-48-159-19:27017")
{ "ok" : 1 }
rs2:PRIMARY> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-01T18:12:26Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-226-133-10:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 210,
			"optime" : {
				"t" : 1362161525,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T18:12:05Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-159-19:27017",
			"health" : 1,
			"state" : 5,
			"stateStr" : "STARTUP2",
			"uptime" : 21,
			"optime" : {
				"t" : 0,
				"i" : 0
			},
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2013-03-01T18:12:25Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : 0,
			"lastHeartbeatMessage" : "initial sync need a member to be primary or secondary to do our initial sync"
		}
	],
	"ok" : 1
}

rs2:PRIMARY> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-03-01T18:12:32Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-226-133-10:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 216,
			"optime" : {
				"t" : 1362161525,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T18:12:05Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-159-19:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 27,
			"optime" : {
				"t" : 1362161525,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-03-01T18:12:05Z"),
			"lastHeartbeat" : ISODate("2013-03-01T18:12:31Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : 1
		}
	],
	"ok" : 1
}
rs2:PRIMARY> rs.addArb("ip-10-229-130-219:30000")
Fri Mar  1 18:13:13.519 DBClientCursor::init call() failed
Fri Mar  1 18:13:13.520 trying reconnect to ip-10-226-133-10:27017
Fri Mar  1 18:13:13.521 reconnect ip-10-226-133-10:27017 failed couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 18:13:13.522 JavaScript execution failed: Error: socket exception [CONNECT_ERROR] for ip-10-226-133-10:27017 at src/mongo/shell/query.js:L78

Comment by Gregor Macadam [ 01/Mar/13 ]

arbiter

ubuntu@ip-10-229-130-219:~$ ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3 --dbpath ./data/ --port 30000 --replSet rs2 --smallfiles
Fri Mar  1 17:43:05.582 [initandlisten] MongoDB starting : pid=8656 port=30000 dbpath=./data/ 64-bit host=ip-10-229-130-219
Fri Mar  1 17:43:05.582 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 17:43:05.582 [initandlisten] git version: 1f7902206e31305e336c155166a6c12ae0f72ab3
Fri Mar  1 17:43:05.582 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 17:43:05.583 [initandlisten] allocator: tcmalloc
Fri Mar  1 17:43:05.583 [initandlisten] options: { dbpath: "./data/", port: 30000, replSet: "rs2", smallfiles: true }
Fri Mar  1 17:43:05.622 [initandlisten] journal dir=./data/journal
Fri Mar  1 17:43:05.622 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 17:43:06.322 [initandlisten] preallocateIsFaster=true 6.7
Fri Mar  1 17:43:07.038 [initandlisten] preallocateIsFaster=true 7.46
Fri Mar  1 17:43:08.806 [initandlisten] preallocateIsFaster=true 7.22
Fri Mar  1 17:43:08.806 [initandlisten] preallocateIsFaster check took 3.183 secs
Fri Mar  1 17:43:08.806 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 17:43:13.390 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 17:43:17.750 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 17:43:22.338 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 17:43:22.338 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 17:43:22.354 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.007 secs
Fri Mar  1 17:43:22.354 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 17:43:22.366 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.008 secs
Fri Mar  1 17:43:22.396 [initandlisten] waiting for connections on port 30000
Fri Mar  1 17:43:22.396 [websvr] admin web console waiting for connections on port 31000
Fri Mar  1 17:43:22.405 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:22.405 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 17:43:32.406 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:42.406 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:52.407 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:02.407 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:12.407 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:22.408 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:32.408 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:42.409 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:52.409 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:02.410 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:12.410 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:22.410 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:32.411 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:42.411 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:52.412 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:02.412 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:12.413 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:22.413 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:32.414 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:42.414 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:52.414 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:02.415 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:12.415 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:22.416 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:32.416 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:42.417 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:52.417 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:02.418 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:12.418 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:22.419 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:32.419 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:42.420 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:48:52.420 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:02.420 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:12.421 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:22.421 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:32.422 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:42.422 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:49:52.423 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:02.423 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:12.423 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:17.146 [initandlisten] connection accepted from 10.226.133.10:50315 #1 (1 connection now open)
Fri Mar  1 17:50:22.424 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:32.424 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:42.425 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:50:47.212 [conn1] end connection 10.226.133.10:50315 (0 connections now open)
Fri Mar  1 17:50:52.425 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:02.426 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:12.426 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:22.426 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:32.427 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:42.427 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:51:52.428 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:02.428 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:12.428 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:22.429 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:32.429 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:42.430 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:52:52.430 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:02.431 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:12.431 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:22.431 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:32.432 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:42.432 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:53:52.433 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:02.433 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:12.434 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:22.434 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:32.434 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:42.435 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:54:52.435 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:02.436 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:12.436 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:22.437 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:32.437 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:42.437 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:55:52.438 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:02.438 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:12.439 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:22.439 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:32.439 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:42.440 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:56:52.440 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:02.441 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:12.441 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:22.442 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:32.442 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:42.442 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:57:52.443 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:02.443 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:12.444 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:22.444 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:32.444 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:42.445 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:58:52.445 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:02.446 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:12.446 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:22.447 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:32.447 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:42.447 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:59:52.448 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:02.448 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:12.449 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:22.449 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:32.450 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:42.450 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:00:52.450 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:02.451 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:12.451 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:22.452 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:32.452 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:42.453 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:01:52.453 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:02:02.454 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 18:02:12.454 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
^CFri Mar  1 18:02:17.737 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 18:02:17.737 [interruptThread] now exiting
Fri Mar  1 18:02:17.737 dbexit: 
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 18:02:17.738 [interruptThread] closing listening socket: 9
Fri Mar  1 18:02:17.738 [interruptThread] closing listening socket: 10
Fri Mar  1 18:02:17.738 [interruptThread] closing listening socket: 11
Fri Mar  1 18:02:17.738 [interruptThread] removing socket file: /tmp/mongodb-30000.sock
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 18:02:17.738 [interruptThread] shutdown: final commit...
Fri Mar  1 18:02:17.739 [interruptThread] shutdown: closing all files...
Fri Mar  1 18:02:17.739 [interruptThread] closeAllFiles() finished
Fri Mar  1 18:02:17.740 [interruptThread] journalCleanup...
Fri Mar  1 18:02:17.740 [interruptThread] removeJournalFiles
Fri Mar  1 18:02:17.771 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 18:02:17.772 dbexit: really exiting now

Comment by Gregor Macadam [ 01/Mar/13 ]

secondary

Fri Mar  1 17:56:34.163 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:56:34.163 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:56:35.157 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:35.158 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:35.159 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:35.160 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:35.161 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:35.162 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:37.163 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:37.164 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:37.165 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:37.166 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:37.167 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:37.168 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:39.170 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:39.171 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:39.172 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:39.173 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:39.174 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:39.174 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:41.176 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:41.177 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:41.178 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:41.178 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:41.179 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:41.180 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.181 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.182 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.183 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:43.186 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.187 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.188 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:43.188 [rsMgr] replSet I don't see a primary and I can't elect myself
Fri Mar  1 17:56:44.167 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:56:44.168 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:56:44.168 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:56:45.189 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:45.191 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:45.192 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:45.192 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:45.193 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:45.194 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:47.196 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:47.197 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:47.198 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:47.199 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:47.199 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:47.200 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:49.202 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:49.203 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:49.204 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:49.205 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:49.206 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:49.207 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:51.209 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:51.209 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:51.210 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:51.211 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:51.212 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:51.213 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:53.214 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:53.215 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:53.216 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:53.217 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:53.218 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:53.219 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:54.172 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:56:54.172 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:56:54.173 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:56:55.220 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:55.221 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:55.222 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:55.223 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:55.224 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:55.225 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:55.225 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:57.227 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:57.228 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:57.229 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:57.229 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:57.230 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:57.231 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:59.232 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:59.233 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:59.234 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:56:59.235 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:59.236 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:56:59.236 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:01.239 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:01.241 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:01.242 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:01.243 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:01.244 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:01.245 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:03.246 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:03.248 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:03.248 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:03.249 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:03.250 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:03.250 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:04.176 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:04.177 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:04.177 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:05.252 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:05.253 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:05.253 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:05.254 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:05.255 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:05.256 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:07.257 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:07.258 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:07.259 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:07.260 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:07.261 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:07.261 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:09.263 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:09.264 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:09.265 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:09.266 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:09.266 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:09.268 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:11.269 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:11.270 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:11.271 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:11.272 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:11.273 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:11.273 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:13.275 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:13.276 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:13.277 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:13.277 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:13.278 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:13.279 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:14.181 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:14.182 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:14.182 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:15.283 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:15.288 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:15.291 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:15.292 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:15.292 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:15.293 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:17.294 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:17.295 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:17.297 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:17.297 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:17.298 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:17.299 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:19.301 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:19.302 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:19.302 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:19.303 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:19.304 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:19.305 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:21.306 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:21.307 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:21.308 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:21.309 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:21.310 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:21.311 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:23.314 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:23.318 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:23.319 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:23.320 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:23.321 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:23.322 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:24.185 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:24.186 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:24.186 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:25.323 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:25.324 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:25.325 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:25.326 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:25.327 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:25.327 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:25.328 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:26.976 [rsSync] replSet syncThread: 14031 Can't take a write lock while out of disk space
Fri Mar  1 17:57:27.330 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:27.331 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:27.332 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:27.333 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:27.333 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:27.334 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:29.336 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:29.337 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:29.338 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:29.339 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:29.340 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:29.341 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:31.343 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:31.344 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:31.349 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:31.353 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:31.358 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:31.362 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:33.364 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:33.365 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:33.366 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:33.367 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:33.367 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:33.368 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:34.190 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:34.191 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:34.191 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:35.369 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:35.371 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:35.373 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:35.375 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:35.377 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:35.380 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:37.381 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:37.382 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:37.383 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:37.384 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:37.385 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:37.386 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:39.387 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:39.389 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:39.391 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:39.394 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:39.396 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:39.398 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:41.399 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:41.400 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:41.401 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:41.402 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:41.402 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:41.403 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:43.405 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:43.406 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:43.407 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:43.408 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:43.409 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:43.409 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:44.195 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:44.196 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:44.196 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:45.411 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:45.412 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:45.412 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:45.413 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:45.414 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:45.415 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:47.416 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:47.417 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:47.418 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:47.419 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:47.420 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:47.421 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:49.422 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:49.423 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:49.424 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:49.425 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:49.426 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:49.427 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:51.429 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:51.430 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:51.431 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:51.432 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:51.432 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:51.433 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:53.435 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:53.436 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:53.437 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:53.437 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:53.438 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:53.439 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:54.200 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:57:54.201 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:57:54.201 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:57:55.441 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:55.442 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:55.443 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:55.443 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:55.444 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:55.445 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:55.446 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:57.448 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:57.449 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:57.450 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:57.451 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:57.452 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:57.452 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:59.454 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:59.455 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:59.455 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:57:59.456 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:59.457 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:57:59.457 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:01.459 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:01.460 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:01.461 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:01.461 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:01.462 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:01.463 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:03.466 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:03.467 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:03.468 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:03.469 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:03.470 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:03.470 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:04.205 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:04.205 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:04.205 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:05.472 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:05.473 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:05.473 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:05.474 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:05.475 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:05.475 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:07.477 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:07.478 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:07.479 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:07.480 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:07.480 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:07.481 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:09.483 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:09.484 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:09.485 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:09.485 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:09.486 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:09.487 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:11.488 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:11.489 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:11.490 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:11.491 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:11.492 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:11.498 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.499 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.500 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.500 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:13.501 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.502 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.503 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:13.503 [rsMgr] replSet I don't see a primary and I can't elect myself
Fri Mar  1 17:58:14.209 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:14.210 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:14.210 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:15.504 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:15.507 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:15.510 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:15.513 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:15.516 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:15.516 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:17.518 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:17.519 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:17.520 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:17.520 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:17.521 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:17.522 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:19.523 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:19.524 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:19.525 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:19.527 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:19.530 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:19.531 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:21.532 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:21.533 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:21.534 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:21.535 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:21.535 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:21.536 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:23.537 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:23.538 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:23.539 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:23.540 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:23.540 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:23.541 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:24.214 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:24.215 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:24.215 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:25.543 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:25.543 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:25.544 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:25.545 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:25.546 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:25.546 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:25.547 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:27.548 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:27.550 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:27.551 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:27.551 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:27.552 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:27.553 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:29.554 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:29.555 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:29.556 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:29.557 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:29.558 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:29.558 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:31.560 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:31.561 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:31.562 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:31.563 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:31.564 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:31.565 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:32.978 [rsSync] replSet syncThread: 14031 Can't take a write lock while out of disk space
Fri Mar  1 17:58:33.566 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:33.567 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:33.568 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:33.569 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:33.570 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:33.571 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:34.219 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:34.219 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:34.219 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:35.572 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:35.573 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:35.574 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:35.574 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:35.575 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:35.576 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:37.577 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:37.578 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:37.579 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:37.580 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:37.580 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:37.581 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:39.583 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:39.584 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:39.585 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:39.586 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:39.586 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:39.587 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:41.589 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:41.590 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:41.591 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:41.592 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:41.592 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:41.593 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:43.594 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:43.595 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:43.596 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:43.597 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:43.598 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:43.598 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:44.223 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:44.224 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:44.224 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:45.600 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:45.601 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:45.602 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:45.603 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:45.604 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:45.604 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:47.606 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:47.607 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:47.608 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:47.608 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:47.609 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:47.610 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:49.611 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:49.612 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:49.613 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:49.613 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:49.615 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:49.615 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:51.617 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:51.618 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:51.618 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:51.619 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:51.620 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:51.620 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:53.622 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:53.622 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:53.623 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:53.624 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:53.625 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:53.626 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:54.228 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:58:54.228 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:58:54.228 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:58:55.627 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:55.628 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:55.629 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:55.629 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:55.630 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:55.631 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:55.632 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:57.633 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:57.634 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:57.635 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:57.635 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:57.636 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:57.637 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:59.638 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:59.639 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:59.640 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:58:59.640 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:59.641 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:58:59.642 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:01.643 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:01.644 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:01.645 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:01.646 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:01.647 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:01.648 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:03.650 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:03.652 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:03.653 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:03.654 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:03.655 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:03.656 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:04.232 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:59:04.233 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:59:04.233 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:59:05.658 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:05.659 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:05.660 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:05.660 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:05.662 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:05.662 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:07.664 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:07.667 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:07.667 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:07.668 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:07.669 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:07.670 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:09.671 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:09.672 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:09.673 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:09.673 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:09.674 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:09.675 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:11.677 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:11.678 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:11.679 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:11.680 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:11.681 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:11.682 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:13.683 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:13.684 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:13.685 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:13.685 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:13.686 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:13.687 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:14.237 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:59:14.237 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:59:14.238 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:59:15.688 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:15.689 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:15.690 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:15.691 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:15.692 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:15.693 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:17.694 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:17.696 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:17.696 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:17.697 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:17.698 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:17.699 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:19.701 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:19.702 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:19.703 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:19.705 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:19.707 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:19.708 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:21.710 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:21.711 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:21.712 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:21.712 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:21.713 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:21.714 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:23.716 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:23.717 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:23.718 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:23.719 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:23.723 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:23.724 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:24.243 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:59:24.244 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:59:24.244 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:59:25.725 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:25.726 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:25.727 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:25.728 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:25.728 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:25.729 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:25.730 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:27.732 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:27.733 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:27.733 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:27.734 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:27.735 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:27.736 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:29.737 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:29.738 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:29.739 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:29.739 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:29.740 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:29.741 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:31.742 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:31.743 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:31.744 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:31.744 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:31.745 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:31.746 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:33.748 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:33.748 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:33.749 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:33.750 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:33.751 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:33.751 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:34.248 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:59:34.248 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:59:34.248 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
Fri Mar  1 17:59:35.753 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:35.755 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:35.756 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:35.757 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:35.759 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:35.760 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:37.762 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:37.763 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:37.763 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:37.764 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:37.765 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:37.766 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:38.979 [rsSync] replSet syncThread: 14031 Can't take a write lock while out of disk space
Fri Mar  1 17:59:39.768 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:39.769 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:39.769 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:39.770 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:39.771 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:39.772 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:41.774 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:41.775 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:41.775 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:41.776 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:41.777 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:41.778 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.779 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.781 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.782 [rsHealthPoll] replset info ip-10-226-133-10:27017 heartbeat failed, retrying
Fri Mar  1 17:59:43.783 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.784 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.785 [rsHealthPoll] couldn't connect to ip-10-226-133-10:27017: couldn't connect to server ip-10-226-133-10:27017
Fri Mar  1 17:59:43.785 [rsMgr] replSet I don't see a primary and I can't elect myself
Fri Mar  1 17:59:44.252 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:59:44.253 [FileAllocator] FileAllocator: posix_fallocate failed: errno:28 No space left on device falling back
Fri Mar  1 17:59:44.253 [FileAllocator] error: failed to allocate new file: ./data/local.2 size: 536608768 failure creating new datafile; lseek failed for fd 12 with errno: errno:2 No such file or directory.  will try again in 10 seconds
^CFri Mar  1 17:59:44.318 got signal 2 (Interrupt), will terminate after current cmd ends
Fri Mar  1 17:59:44.318 [interruptThread] now exiting
Fri Mar  1 17:59:44.318 dbexit: 
Fri Mar  1 17:59:44.318 [interruptThread] shutdown: going to close listening sockets...
Fri Mar  1 17:59:44.318 [interruptThread] closing listening socket: 9
Fri Mar  1 17:59:44.318 [interruptThread] closing listening socket: 10
Fri Mar  1 17:59:44.318 [interruptThread] closing listening socket: 11
Fri Mar  1 17:59:44.319 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: going to flush diaglog...
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: going to close sockets...
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: waiting for fs preallocator...
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: lock for final commit...
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: final commit...
Fri Mar  1 17:59:44.319 [interruptThread] shutdown: closing all files...
Fri Mar  1 17:59:44.319 [interruptThread] closeAllFiles() finished
Fri Mar  1 17:59:44.319 [interruptThread] journalCleanup...
Fri Mar  1 17:59:44.319 [interruptThread] removeJournalFiles
Fri Mar  1 17:59:44.355 [interruptThread] shutdown: removing fs lock...
Fri Mar  1 17:59:44.355 dbexit: really exiting now

Comment by Gregor Macadam [ 01/Mar/13 ]

ubuntu@ip-10-226-133-10:~$ ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3 --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ./gregorFreeBSD.pem --sslCAFile ./cacert.pem --smallfiles --replSet rs2
Fri Mar  1 17:41:42.826 [initandlisten] MongoDB starting : pid=10886 port=27017 dbpath=./data/ 64-bit host=ip-10-226-133-10
Fri Mar  1 17:41:42.826 [initandlisten] db version v2.4.0-rc2-pre-, pdfile version 4.5
Fri Mar  1 17:41:42.826 [initandlisten] git version: 1f7902206e31305e336c155166a6c12ae0f72ab3
Fri Mar  1 17:41:42.826 [initandlisten] build info: Linux ip-10-36-129-84 3.2.0-36-virtual #57-Ubuntu SMP Tue Jan 8 22:04:49 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
Fri Mar  1 17:41:42.826 [initandlisten] allocator: tcmalloc
Fri Mar  1 17:41:42.826 [initandlisten] options: { dbpath: "./data/", replSet: "rs2", smallfiles: true, sslCAFile: "./cacert.pem", sslOnNormalPorts: true, sslPEMKeyFile: "./gregorFreeBSD.pem" }
Fri Mar  1 17:41:42.861 [initandlisten] journal dir=./data/journal
Fri Mar  1 17:41:42.861 [initandlisten] recover : no journal files present, no recovery needed
Fri Mar  1 17:41:43.685 [initandlisten] preallocateIsFaster=true 8.4
Fri Mar  1 17:41:44.513 [initandlisten] preallocateIsFaster=true 8.3
Fri Mar  1 17:41:46.337 [initandlisten] preallocateIsFaster=true 9.08
Fri Mar  1 17:41:46.337 [initandlisten] preallocateIsFaster check took 3.475 secs
Fri Mar  1 17:41:46.338 [initandlisten] preallocating a journal file ./data/journal/prealloc.0
Fri Mar  1 17:41:50.301 [initandlisten] preallocating a journal file ./data/journal/prealloc.1
Fri Mar  1 17:41:54.497 [initandlisten] preallocating a journal file ./data/journal/prealloc.2
Fri Mar  1 17:41:58.887 [FileAllocator] allocating new datafile ./data/local.ns, filling with zeroes...
Fri Mar  1 17:41:58.887 [FileAllocator] creating directory ./data/_tmp
Fri Mar  1 17:41:58.901 [FileAllocator] done allocating datafile ./data/local.ns, size: 16MB,  took 0.008 secs
Fri Mar  1 17:41:58.901 [FileAllocator] allocating new datafile ./data/local.0, filling with zeroes...
Fri Mar  1 17:41:58.909 [FileAllocator] done allocating datafile ./data/local.0, size: 16MB,  took 0.007 secs
Fri Mar  1 17:41:58.914 [initandlisten] waiting for connections on port 27017 ssl
Fri Mar  1 17:41:58.915 [websvr] admin web console waiting for connections on port 28017 ssl
Fri Mar  1 17:41:58.922 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:41:58.922 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Fri Mar  1 17:42:08.923 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:42:18.923 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:42:28.924 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:42:38.924 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:42:48.925 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:42:58.925 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:08.926 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:18.926 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:28.927 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:38.927 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:48.928 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:43:58.928 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:08.929 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:18.929 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:28.930 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:38.930 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:48.930 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:44:58.931 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:08.931 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:18.932 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:28.932 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:38.933 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:48.933 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:45:58.934 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:08.934 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:18.935 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:28.935 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:38.936 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:48.936 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:46:58.937 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:08.937 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:18.938 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:28.938 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:37.317 [initandlisten] connection accepted from 10.227.21.162:44368 #1 (1 connection now open)
Fri Mar  1 17:47:38.939 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Fri Mar  1 17:47:43.480 [conn1] replSet replSetInitiate admin command received from client
Fri Mar  1 17:47:43.481 [conn1] replSet info initiate : no configuration specified.  Using a default configuration for the set
Fri Mar  1 17:47:43.482 [conn1] replSet created this configuration for initiation : { _id: "rs2", members: [ { _id: 0, host: "ip-10-226-133-10:27017" } ] }
Fri Mar  1 17:47:43.482 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Fri Mar  1 17:47:43.539 [conn1] replSet replSetInitiate all members seem up
Fri Mar  1 17:47:43.540 [conn1] ******
Fri Mar  1 17:47:43.540 [conn1] creating replication oplog of size: 990MB...
Fri Mar  1 17:47:43.540 [FileAllocator] allocating new datafile ./data/local.1, filling with zeroes...
Fri Mar  1 17:47:43.553 [FileAllocator] done allocating datafile ./data/local.1, size: 511MB,  took 0.012 secs
Fri Mar  1 17:47:43.553 [FileAllocator] allocating new datafile ./data/local.2, filling with zeroes...
Fri Mar  1 17:47:43.561 [FileAllocator] done allocating datafile ./data/local.2, size: 511MB,  took 0.007 secs
Fri Mar  1 17:47:43.561 [conn1] ******
Fri Mar  1 17:47:43.562 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 17:47:43.562 [conn1] replSet saveConfigLocally done
Fri Mar  1 17:47:43.562 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
Fri Mar  1 17:47:48.939 [rsStart] replSet I am ip-10-226-133-10:27017
Fri Mar  1 17:47:48.940 [rsStart] replSet STARTUP2
Fri Mar  1 17:47:49.949 [rsSync] replSet SECONDARY
Fri Mar  1 17:47:49.950 [rsMgr] replSet info electSelf 0
Fri Mar  1 17:47:49.955 [rsMgr] replSet PRIMARY
Fri Mar  1 17:49:08.586 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Fri Mar  1 17:49:08.697 [conn1] replSet replSetReconfig [2]
Fri Mar  1 17:49:08.697 [conn1] replSet info saving a newer config version to local.system.replset
Fri Mar  1 17:49:08.699 [conn1] replSet saveConfigLocally done
Fri Mar  1 17:49:08.699 [conn1] replSet info : additive change to configuration
Fri Mar  1 17:49:08.699 [conn1] replSet replSetReconfig new config saved locally
Fri Mar  1 17:49:08.699 [conn1] command admin.$cmd command: { replSetReconfig: { _id: "rs2", version: 2, members: [ { _id: 0, host: "ip-10-226-133-10:27017" }, { _id: 1.0, host: "ip-10-48-159-19:27017" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1524 reslen:37 113ms
Fri Mar  1 17:49:08.700 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is up
Fri Mar  1 17:49:08.700 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Fri Mar  1 17:49:16.325 [initandlisten] connection accepted from 10.48.159.19:55320 #2 (2 connections now open)
Fri Mar  1 17:49:16.335 [initandlisten] connection accepted from 10.48.159.19:55321 #3 (3 connections now open)
Fri Mar  1 17:49:16.336 [conn2] end connection 10.48.159.19:55320 (2 connections now open)
Fri Mar  1 17:49:16.705 [rsHealthPoll] replset info ip-10-48-159-19:27017 thinks that we are down
Fri Mar  1 17:49:16.705 [rsHealthPoll] replSet member ip-10-48-159-19:27017 is now in state STARTUP2
Fri Mar  1 17:49:46.409 [conn3] end connection 10.48.159.19:55321 (1 connection now open)
Fri Mar  1 17:49:46.413 [initandlisten] connection accepted from 10.48.159.19:55322 #4 (2 connections now open)
Fri Mar  1 17:50:13.838 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
Fri Mar  1 17:50:16.442 [conn4] end connection 10.48.159.19:55322 (1 connection now open)
Fri Mar  1 17:50:16.443 [initandlisten] connection accepted from 10.48.159.19:55323 #5 (2 connections now open)
Fri Mar  1 17:50:43.840 [conn1]   Fatal Assertion 16697
0xb856b3 0xb4925d 0xb75f8d 0xb781d4 0xb733a1 0x74af5d 0x74b2a8 0xa20c8a 0xa231af 0xa2cb1b 0xa526bd 0xa38814 0xa38ac5 0x855edf 0x856821 0x857967 0x96841a 0x96b3c1 0x91553c 0x71a2f9 
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo15printStackTraceERSo+0x23) [0xb856b3]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo13fassertFailedEi+0x4d) [0xb4925d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb75f8d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager7connectEi+0x24) [0xb781d4]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb733a1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74af5d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74b2a8]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa20c8a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa231af]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa2cb1b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa526bd]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa38814]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa38ac5]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x855edf]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x856821]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x857967]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0x96841a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xcf1) [0x96b3c1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xb4c) [0x91553c]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x89) [0x71a2f9]
Fri Mar  1 17:50:43.866 [conn1] 
 
***aborting after fassert() failure
 
 
Fri Mar  1 17:50:43.866 Got signal: 6 (Aborted).
 
Fri Mar  1 17:50:43.872 Backtrace:
0xb856b3 0x70921c 0x7f90548ce4a0 0x7f90548ce425 0x7f90548d1b8b 0xb49298 0xb75f8d 0xb781d4 0xb733a1 0x74af5d 0x74b2a8 0xa20c8a 0xa231af 0xa2cb1b 0xa526bd 0xa38814 0xa38ac5 0x855edf 0x856821 0x857967 
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo15printStackTraceERSo+0x23) [0xb856b3]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10abruptQuitEi+0x3fc) [0x70921c]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f90548ce4a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f90548ce425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f90548d1b8b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo13fassertFailedEi+0x88) [0xb49298]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb75f8d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10SSLManager7connectEi+0x24) [0xb781d4]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb733a1]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74af5d]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74b2a8]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa20c8a]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa231af]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa2cb1b]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa526bd]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa38814]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa38ac5]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x855edf]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x856821]
 ./mongod_1f7902206e31305e336c155166a6c12ae0f72ab3(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x857967]
 

Comment by auto [ 01/Mar/13 ]

Author:

{u'date': u'2013-03-01T15:06:18Z', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-8734 use a recursive mutex to work around OpenSSL deadlock
Branch: master
https://github.com/mongodb/mongo/commit/1f7902206e31305e336c155166a6c12ae0f72ab3

Comment by Eric Milkie [ 01/Mar/13 ]

Original report (via openssl-dev)
http://www.mail-archive.com/openssl-dev@openssl.org/msg30970.html

Comment by Eric Milkie [ 01/Mar/13 ]

Here is the official bug report, with fix.
http://cvs.openssl.org/chngview?cn=22568

Comment by Gregor Macadam [ 28/Feb/13 ]

Nice catch!

Comment by Eric Milkie [ 28/Feb/13 ]

Believe it or not, this is caused by this bug:
https://github.com/openssl/openssl/commit/4baee3031cf5529f519756b7a30bd5b078980c74#crypto/asn1/x_pubkey.c

The bug affects all three versions of OpenSSL, 1.0.1, 1.0.0, and 0.9.8.
On February 5, 2013, new releases of all three versions appeared, so if you upgrade to 1.0.1d, 1.0.0k, or 0.9.8y, this problem will be fixed.
I think I will try to add a mutex in our code around certain calls into the OpenSSL library to avoid hitting this bug.

Comment by Gregor Macadam [ 26/Feb/13 ]

Just did fresh repro

ubuntu@ip-10-226-183-211:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.4.0-rc1$ ./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslCAFile ../cacert.pem --smallfiles --replSet rs2

ubuntu@ip-10-48-130-227:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.4.0-rc1$ ./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslCAFile ../cacert.pem --smallfiles --replSet rs2

ubuntu@ip-10-48-130-227:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.4.0-rc1$ ./bin/mongod --dbpath ./data/arb/ --port 30000 --replSet rs2

Steps to repro:

ubuntu@ip-10-36-133-56:~/mongodb-linux-x86_64-subscription-ubuntu1104-2.4.0-rc1$ ./bin/mongo --host ip-10-226-183-211 --ssl --sslPEMKeyFile ../sslCA/gregorFreeBSD.pem 

rs.initiate()

rs2:SECONDARY> rs.add("ip-10-48-130-227:27017")
{ "ok" : 1 }

rs2:PRIMARY> rs.status()
{
	"set" : "rs2",
	"date" : ISODate("2013-02-26T20:50:10Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-226-183-211:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 411,
			"optime" : {
				"t" : 1361911493000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-26T20:44:53Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-130-227:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 317,
			"optime" : {
				"t" : 1361911493000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-26T20:44:53Z"),
			"lastHeartbeat" : ISODate("2013-02-26T20:50:09Z"),
			"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
			"pingMs" : 1
		}
	],
	"ok" : 1
}

Mongo shell hangs for 30 seconds when adding arbiter ....

rs2:PRIMARY> rs.addArb("ip-10-48-130-227:30000")
Tue Feb 26 20:52:41.559 DBClientCursor::init call() failed
Tue Feb 26 20:52:41.560 trying reconnect to ip-10-226-183-211:27017
Tue Feb 26 20:52:41.561 reconnect ip-10-226-183-211:27017 failed couldn't connect to server ip-10-226-183-211:27017
Tue Feb 26 20:52:41.561 JavaScript execution failed: Error: socket exception [CONNECT_ERROR] for ip-10-226-183-211:27017 at src/mongo/shell/query.js:L78
> 

...before primary goes down

Tue Feb 26 20:44:01.341 [rsMgr] replSet PRIMARY
Tue Feb 26 20:44:53.036 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Tue Feb 26 20:44:53.094 [conn1] replSet replSetReconfig [2]
Tue Feb 26 20:44:53.094 [conn1] replSet info saving a newer config version to local.system.replset
Tue Feb 26 20:44:53.146 [conn1] replSet saveConfigLocally done
Tue Feb 26 20:44:53.146 [conn1] replSet info : additive change to configuration
Tue Feb 26 20:44:53.147 [conn1] replSet replSetReconfig new config saved locally
Tue Feb 26 20:44:53.147 [conn1] command admin.$cmd command: { replSetReconfig: { _id: "rs2", version: 2, members: [ { _id: 0, host: "ip-10-226-183-211:27017" }, { _id: 1.0, host: "ip-10-48-130-227:27017" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:51742 reslen:37 110ms
Tue Feb 26 20:44:53.148 [rsHealthPoll] replSet member ip-10-48-130-227:27017 is up
Tue Feb 26 20:44:53.148 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Tue Feb 26 20:44:56.924 [initandlisten] connection accepted from 10.48.130.227:47932 #2 (2 connections now open)
Tue Feb 26 20:44:57.177 [initandlisten] connection accepted from 10.48.130.227:47933 #3 (3 connections now open)
Tue Feb 26 20:44:57.181 [conn2] end connection 10.48.130.227:47932 (1 connection now open)
Tue Feb 26 20:44:59.618 [rsHealthPoll] replSet member ip-10-48-130-227:27017 is now in state STARTUP2
Tue Feb 26 20:45:13.622 [initandlisten] connection accepted from 10.48.130.227:47934 #4 (3 connections now open)
Tue Feb 26 20:45:13.702 [conn4] end connection 10.48.130.227:47934 (2 connections now open)
Tue Feb 26 20:45:14.569 [initandlisten] connection accepted from 10.48.130.227:47935 #5 (3 connections now open)
Tue Feb 26 20:45:14.704 [initandlisten] connection accepted from 10.48.130.227:47936 #6 (4 connections now open)
Tue Feb 26 20:45:15.635 [rsHealthPoll] replSet member ip-10-48-130-227:27017 is now in state RECOVERING
Tue Feb 26 20:45:15.733 [slaveTracking] build index local.slaves { _id: 1 }
Tue Feb 26 20:45:15.735 [slaveTracking] build index done.  scanned 0 total records. 0.001 secs
Tue Feb 26 20:45:17.636 [rsHealthPoll] replSet member ip-10-48-130-227:27017 is now in state SECONDARY
Tue Feb 26 20:45:25.673 [conn3] end connection 10.48.130.227:47933 (3 connections now open)
Tue Feb 26 20:45:25.676 [initandlisten] connection accepted from 10.48.130.227:47937 #7 (4 connections now open)
Tue Feb 26 20:45:55.701 [conn7] end connection 10.48.130.227:47937 (3 connections now open)
Tue Feb 26 20:45:55.703 [initandlisten] connection accepted from 10.48.130.227:47938 #8 (4 connections now open)
Tue Feb 26 20:46:25.734 [conn8] end connection 10.48.130.227:47938 (3 connections now open)
Tue Feb 26 20:46:25.736 [initandlisten] connection accepted from 10.48.130.227:47939 #9 (4 connections now open)
Tue Feb 26 20:46:55.756 [conn9] end connection 10.48.130.227:47939 (3 connections now open)
Tue Feb 26 20:46:55.757 [initandlisten] connection accepted from 10.48.130.227:47940 #10 (4 connections now open)
Tue Feb 26 20:47:25.779 [conn10] end connection 10.48.130.227:47940 (3 connections now open)
Tue Feb 26 20:47:25.780 [initandlisten] connection accepted from 10.48.130.227:47941 #11 (4 connections now open)
Tue Feb 26 20:47:55.805 [conn11] end connection 10.48.130.227:47941 (3 connections now open)
Tue Feb 26 20:47:55.806 [initandlisten] connection accepted from 10.48.130.227:47942 #12 (4 connections now open)
Tue Feb 26 20:48:25.828 [conn12] end connection 10.48.130.227:47942 (3 connections now open)
Tue Feb 26 20:48:25.829 [initandlisten] connection accepted from 10.48.130.227:47943 #13 (4 connections now open)
Tue Feb 26 20:48:55.849 [conn13] end connection 10.48.130.227:47943 (3 connections now open)
Tue Feb 26 20:48:55.850 [initandlisten] connection accepted from 10.48.130.227:47944 #14 (4 connections now open)
Tue Feb 26 20:49:25.869 [conn14] end connection 10.48.130.227:47944 (3 connections now open)
Tue Feb 26 20:49:25.870 [initandlisten] connection accepted from 10.48.130.227:47945 #15 (4 connections now open)
Tue Feb 26 20:49:55.902 [conn15] end connection 10.48.130.227:47945 (3 connections now open)
Tue Feb 26 20:49:55.903 [initandlisten] connection accepted from 10.48.130.227:47946 #16 (4 connections now open)
Tue Feb 26 20:50:25.966 [conn16] end connection 10.48.130.227:47946 (3 connections now open)
Tue Feb 26 20:50:26.150 [initandlisten] connection accepted from 10.48.130.227:47947 #17 (4 connections now open)
Tue Feb 26 20:50:56.206 [conn17] end connection 10.48.130.227:47947 (3 connections now open)
Tue Feb 26 20:50:56.372 [initandlisten] connection accepted from 10.48.130.227:47948 #18 (4 connections now open)
Tue Feb 26 20:51:26.403 [conn18] end connection 10.48.130.227:47948 (3 connections now open)
Tue Feb 26 20:51:26.583 [initandlisten] connection accepted from 10.48.130.227:47949 #19 (4 connections now open)
Tue Feb 26 20:51:45.129 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
Tue Feb 26 20:51:56.613 [conn19] end connection 10.48.130.227:47949 (3 connections now open)
Tue Feb 26 20:51:56.614 [initandlisten] connection accepted from 10.48.130.227:47950 #20 (4 connections now open)
Tue Feb 26 20:52:15.132 [conn1]   Fatal Assertion 16697
0xb94363 0xb57bfd 0xb84e3d 0xb87084 0xb82261 0x74e8ed 0x74ec38 0xa2fdaa 0xa322cf 0xa3bc3b 0xa617fd 0xa47954 0xa47c05 0x85c09f 0x85c9e1 0x85db27 0x97753a 0x97a4e1 0x91b7dc 0x71dc89 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xb94363]
 ./bin/mongod(_ZN5mongo13fassertFailedEi+0x4d) [0xb57bfd]
 ./bin/mongod(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb84e3d]
 ./bin/mongod(_ZN5mongo10SSLManager7connectEi+0x24) [0xb87084]
 ./bin/mongod(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb82261]
 ./bin/mongod(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74e8ed]
 ./bin/mongod(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74ec38]
 ./bin/mongod(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa2fdaa]
 ./bin/mongod(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa322cf]
 ./bin/mongod(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa3bc3b]
 ./bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa617fd]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa47954]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa47c05]
 ./bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x85c09f]
 ./bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x85c9e1]
 ./bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x85db27]
 ./bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0x97753a]
 ./bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xcf1) [0x97a4e1]
 ./bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xb4c) [0x91b7dc]
 ./bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x89) [0x71dc89]
Tue Feb 26 20:52:15.182 [conn1] 
 
***aborting after fassert() failure
 
 
Tue Feb 26 20:52:15.182 Got signal: 6 (Aborted).
 
Tue Feb 26 20:52:15.190 Backtrace:
0xb94363 0x70cb9c 0x7f5fa8a004a0 0x7f5fa8a00425 0x7f5fa8a03b8b 0xb57c38 0xb84e3d 0xb87084 0xb82261 0x74e8ed 0x74ec38 0xa2fdaa 0xa322cf 0xa3bc3b 0xa617fd 0xa47954 0xa47c05 0x85c09f 0x85c9e1 0x85db27 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xb94363]
 ./bin/mongod(_ZN5mongo10abruptQuitEi+0x3fc) [0x70cb9c]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f5fa8a004a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f5fa8a00425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f5fa8a03b8b]
 ./bin/mongod(_ZN5mongo13fassertFailedEi+0x88) [0xb57c38]
 ./bin/mongod(_ZN5mongo10SSLManager12_ssl_connectEP6ssl_st+0x6d) [0xb84e3d]
 ./bin/mongod(_ZN5mongo10SSLManager7connectEi+0x24) [0xb87084]
 ./bin/mongod(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x31) [0xb82261]
 ./bin/mongod(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x74e8ed]
 ./bin/mongod(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x74ec38]
 ./bin/mongod(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa2fdaa]
 ./bin/mongod(_ZN5mongo10ScopedConnC2ERKSs+0xef) [0xa322cf]
 ./bin/mongod(_ZN5mongo16requestHeartbeatERKSsS1_S1_RNS_7BSONObjEiRib+0x2cb) [0xa3bc3b]
 ./bin/mongod(_ZN5mongo29checkMembersUpForConfigChangeERKNS_13ReplSetConfigERNS_14BSONObjBuilderEb+0x28d) [0xa617fd]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig4_runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x4c4) [0xa47954]
 ./bin/mongod(_ZN5mongo18CmdReplSetReconfig3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x75) [0xa47c05]
 ./bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x85c09f]
 ./bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x641) [0x85c9e1]
 ./bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x297) [0x85db27]

Secondary and arbiter stay up.

Comment by Eric Milkie [ 26/Feb/13 ]

I can't reproduce this. What were the command lines you used for the members of the replica set?

Generated at Thu Feb 08 03:18:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.