Home > mci_0.9_linux_64_debug_duroff #1339 > replicasets_0 > initial_sync1.js
Plain Text
2013-10-14 17:28:06 EDT 
MongoDB shell version: 2.5.3-pre-
false
1. Bring up set
ReplSetTest Starting Set
ReplSetTest n is : 0
ReplSetTest n: 0 ports: [ 31000, 31001 ]    31000 number
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31000,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : undefined,
    "pathOpts" : {
        "node" : 0,
        "set" : "jstests_initsync1"
    }
}
ReplSetTest Starting....
Resetting db path '/data/db/jstests_initsync1-0'
2013-10-14T21:28:07.029+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-0 --setParameter enableTestCommands=1
2013-10-14T21:28:07.031+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:28:07.074+0000 ** WARNING: --rest is specified without --httpinterface,
 m31000| 2013-10-14T21:28:07.075+0000 **          enabling http interface
 m31000| note: noprealloc may hurt performance in many applications
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] MongoDB starting : pid=9277 port=31000 dbpath=/data/db/jstests_initsync1-0 64-bit host=ip-10-142-147-116
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] _DEBUG build (which is slower)
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] 
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] **       Not recommended for production.
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] 
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] db version v2.5.3-pre-
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] git version: 9fcdfeb195c42e91a49e37212a65fea624d71c13
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] allocator: tcmalloc
 m31000| 2013-10-14T21:28:07.082+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-0", noprealloc: true, oplogSize: 40, port: 31000, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31000| 2013-10-14T21:28:07.113+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31000| 2013-10-14T21:28:07.113+0000 [initandlisten] opening db:  admin
 m31000| 2013-10-14T21:28:07.114+0000 [initandlisten] 
 m31000| debug have W lock but w would suffice for command create
 m31000| 2013-10-14T21:28:07.114+0000 [initandlisten] opening db:  local
 m31000| 2013-10-14T21:28:07.115+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.ns, filling with zeroes...
 m31000| 2013-10-14T21:28:07.115+0000 [FileAllocator] creating directory /data/db/jstests_initsync1-0/_tmp
 m31000| 2013-10-14T21:28:07.129+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.ns, size: 16MB,  took 0.005 secs
 m31000| 2013-10-14T21:28:07.131+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.0, filling with zeroes...
 m31000| 2013-10-14T21:28:07.134+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.0, size: 16MB,  took 0.003 secs
 m31000| 2013-10-14T21:28:07.134+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1-0/local.0 n:0
 m31000| 2013-10-14T21:28:07.134+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31000| 2013-10-14T21:28:07.134+0000 [initandlisten] waiting for connections on port 31000
 m31000| 2013-10-14T21:28:07.135+0000 [websvr] admin web console waiting for connections on port 32000
 m31000| 2013-10-14T21:28:07.137+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31000| 2013-10-14T21:28:07.137+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31000| 2013-10-14T21:28:07.232+0000 [initandlisten] connection accepted from 127.0.0.1:59075 #1 (1 connection now open)
[ connection to ip-10-142-147-116:31000 ]
ReplSetTest n is : 1
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31001,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : undefined,
    "pathOpts" : {
        "node" : 1,
        "set" : "jstests_initsync1"
    }
}
ReplSetTest Starting....
Resetting db path '/data/db/jstests_initsync1-1'
2013-10-14T21:28:07.243+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-1 --setParameter enableTestCommands=1
2013-10-14T21:28:07.244+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:28:07.288+0000 ** WARNING: --rest is specified without --httpinterface,
 m31001| 2013-10-14T21:28:07.289+0000 **          enabling http interface
 m31001| note: noprealloc may hurt performance in many applications
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] MongoDB starting : pid=9326 port=31001 dbpath=/data/db/jstests_initsync1-1 64-bit host=ip-10-142-147-116
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] _DEBUG build (which is slower)
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] 
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] **       Not recommended for production.
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] 
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] db version v2.5.3-pre-
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] git version: 9fcdfeb195c42e91a49e37212a65fea624d71c13
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] allocator: tcmalloc
 m31001| 2013-10-14T21:28:07.296+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31001| 2013-10-14T21:28:07.300+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31001| 2013-10-14T21:28:07.300+0000 [initandlisten] opening db:  admin
 m31001| 2013-10-14T21:28:07.301+0000 [initandlisten] 
 m31001| debug have W lock but w would suffice for command create
 m31001| 2013-10-14T21:28:07.301+0000 [initandlisten] opening db:  local
 m31001| 2013-10-14T21:28:07.301+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.ns, filling with zeroes...
 m31001| 2013-10-14T21:28:07.301+0000 [FileAllocator] creating directory /data/db/jstests_initsync1-1/_tmp
 m31001| 2013-10-14T21:28:07.306+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.ns, size: 16MB,  took 0.002 secs
 m31001| 2013-10-14T21:28:07.308+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.0, filling with zeroes...
 m31001| 2013-10-14T21:28:07.311+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.0, size: 16MB,  took 0.003 secs
 m31001| 2013-10-14T21:28:07.311+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1-1/local.0 n:0
 m31001| 2013-10-14T21:28:07.311+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31001| 2013-10-14T21:28:07.312+0000 [initandlisten] waiting for connections on port 31001
 m31001| 2013-10-14T21:28:07.312+0000 [websvr] admin web console waiting for connections on port 32001
 m31001| 2013-10-14T21:28:07.314+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31001| 2013-10-14T21:28:07.314+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31001| 2013-10-14T21:28:07.445+0000 [initandlisten] connection accepted from 127.0.0.1:59039 #1 (1 connection now open)
[
    connection to ip-10-142-147-116:31000,
    connection to ip-10-142-147-116:31001
]
{
    "replSetInitiate" : {
        "_id" : "jstests_initsync1",
        "members" : [
            {
                "_id" : 0,
                "host" : "ip-10-142-147-116:31000"
            },
            {
                "_id" : 1,
                "host" : "ip-10-142-147-116:31001"
            }
        ]
    }
}
 m31000| 2013-10-14T21:28:07.451+0000 [conn1] replSet replSetInitiate admin command received from client
 m31000| 2013-10-14T21:28:07.452+0000 [conn1] replSet replSetInitiate config object parses ok, 2 members specified
 m31001| 2013-10-14T21:28:07.454+0000 [initandlisten] connection accepted from 10.142.147.116:36876 #2 (2 connections now open)
 m31000| 2013-10-14T21:28:07.455+0000 [conn1] replSet replSetInitiate all members seem up
 m31000| 2013-10-14T21:28:07.455+0000 [conn1] ******
 m31000| 2013-10-14T21:28:07.455+0000 [conn1] creating replication oplog of size: 40MB...
 m31000| 2013-10-14T21:28:07.455+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.1, filling with zeroes...
 m31000| 2013-10-14T21:28:07.458+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.1, size: 64MB,  took 0.002 secs
 m31000| 2013-10-14T21:28:07.458+0000 [conn1] datafileheader::init initializing /data/db/jstests_initsync1-0/local.1 n:1
 m31000| 2013-10-14T21:28:07.467+0000 [conn1] ******
 m31000| 2013-10-14T21:28:07.467+0000 [conn1] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 1, members: [ { _id: 0, host: "ip-10-142-147-116:31000" }, { _id: 1, host: "ip-10-142-147-116:31001" } ] }
 m31000| 2013-10-14T21:28:07.468+0000 [conn1] replSet saveConfigLocally done
 m31000| 2013-10-14T21:28:07.468+0000 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}
2013-10-14T21:28:05.227+0000 [conn38] CMD: drop test.remove4
2013-10-14T21:27:59.428+0000 [conn28] CMD: drop test.remove6
2013-10-14 17:28:09 EDT 
2013-10-14T21:28:05.228+0000 [conn38] build index on: test.remove4 properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "test.remove4", "name" : "_id_" }
2013-10-14T21:27:59.428+0000 [conn73] mr failed, removing collection :: caused by :: 16712 JavaScript execution terminated
 m31000| 2013-10-14T21:28:08.138+0000 [rsStart] replSet I am ip-10-142-147-116:31000
2013-10-14 17:28:12 EDT 
2013-10-14T21:28:05.229+0000 [conn38] build index done.  scanned 0 total records. 0 secs
2013-10-14T21:27:59.430+0000 [conn28] CMD: drop test.remove6
 m31000| 2013-10-14T21:28:08.138+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31001
2013-10-14 17:28:14 EDT 
2013-10-14T21:28:05.232+0000 [conn38] end connection 127.0.0.1:42533 (0 connections now open)
 m31000| 2013-10-14T21:28:08.138+0000 [rsStart] replSet STARTUP2
2013-10-14 17:28:17 EDT 
 m31000| 2013-10-14T21:28:08.138+0000 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
2013-10-14 17:28:20 EDT 
 m31000| 2013-10-14T21:28:08.138+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is up
2013-10-14 17:28:22 EDT 
 m31000| 2013-10-14T21:28:08.139+0000 [rsSync] replSet SECONDARY
2013-10-14 17:28:25 EDT 
 m31000| 2013-10-14T21:28:08.139+0000 [rsMgr] replSet freshest returns { startupStatus: 3, info: "run rs.initiate(...) if not yet done for the set", ok: 0.0, errmsg: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)" }
2013-10-14 17:28:28 EDT 
 m31000| 2013-10-14T21:28:08.139+0000 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
2013-10-14 17:28:30 EDT 
 m31000| 2013-10-14T21:28:08.141+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
2013-10-14 17:28:32 EDT 
 m31000| 2013-10-14T21:28:08.141+0000 [rsSyncNotifier] have free list for local.$freelist
2013-10-14 17:28:35 EDT 
 m31000| 2013-10-14T21:28:08.143+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
2013-10-14 17:28:37 EDT 
 m31001| 2013-10-14T21:28:08.314+0000 [rsStart] trying to contact ip-10-142-147-116:31000
2013-10-14 17:28:40 EDT 
 m31000| 2013-10-14T21:28:08.316+0000 [initandlisten] connection accepted from 10.142.147.116:58309 #2 (2 connections now open)
2013-10-14 17:28:42 EDT 
 m31001| 2013-10-14T21:28:08.317+0000 [rsStart] replSet load config ok from ip-10-142-147-116:31000
2013-10-14 17:28:45 EDT 
 m31001| 2013-10-14T21:28:08.317+0000 [rsStart] replSet I am ip-10-142-147-116:31001
2013-10-14 17:28:47 EDT 
 m31001| 2013-10-14T21:28:08.317+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31000
2013-10-14 17:28:50 EDT 
 m31001| 2013-10-14T21:28:08.317+0000 [rsStart] replSet got config version 1 from a remote, saving locally
2013-10-14 17:28:52 EDT 
 m31001| 2013-10-14T21:28:08.317+0000 [rsStart] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 1, members: [ { _id: 0, host: "ip-10-142-147-116:31000" }, { _id: 1, host: "ip-10-142-147-116:31001" } ] }
2013-10-14 17:28:55 EDT 
 m31001| 2013-10-14T21:28:08.318+0000 [rsStart] replSet saveConfigLocally done
2013-10-14 17:28:57 EDT 
 m31001| 2013-10-14T21:28:08.318+0000 [rsStart] replSet STARTUP2
2013-10-14 17:29:00 EDT 
 m31001| 2013-10-14T21:28:08.318+0000 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
2013-10-14 17:29:03 EDT 
 m31001| 2013-10-14T21:28:08.318+0000 [rsSync] ******
2013-10-14 17:29:05 EDT 
 m31001| 2013-10-14T21:28:08.318+0000 [rsSync] creating replication oplog of size: 40MB...
2013-10-14 17:29:07 EDT 
 m31001| 2013-10-14T21:28:08.319+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.1, filling with zeroes...
2013-10-14 17:29:15 EDT 
 m31001| 2013-10-14T21:28:08.321+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.1, size: 64MB,  took 0.002 secs
2013-10-14 17:29:18 EDT 
 m31001| 2013-10-14T21:28:08.321+0000 [rsSync] datafileheader::init initializing /data/db/jstests_initsync1-1/local.1 n:1
2013-10-14 17:29:20 EDT 
 m31001| 2013-10-14T21:28:08.332+0000 [rsSync] ******
2013-10-14 17:29:23 EDT 
 m31001| 2013-10-14T21:28:08.332+0000 [rsSync] replSet initial sync pending
2013-10-14 17:29:25 EDT 
 m31001| 2013-10-14T21:28:08.332+0000 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
2013-10-14 17:29:28 EDT 
 m31001| 2013-10-14T21:28:08.334+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
2013-10-14 17:29:30 EDT 
 m31001| 2013-10-14T21:28:08.335+0000 [rsSyncNotifier] have free list for local.$freelist
2013-10-14 17:29:33 EDT 
 m31001| 2013-10-14T21:28:08.336+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
2013-10-14 17:29:35 EDT 
 m31000| 2013-10-14T21:28:10.139+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 thinks that we are down
2013-10-14 17:29:38 EDT 
 m31000| 2013-10-14T21:28:10.139+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state STARTUP2
2013-10-14 17:29:40 EDT 
 m31000| 2013-10-14T21:28:10.139+0000 [rsMgr] not electing self, ip-10-142-147-116:31001 would veto with 'I don't think ip-10-142-147-116:31000 is electable'
2013-10-14 17:29:43 EDT 
 m31001| 2013-10-14T21:28:10.318+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is up
2013-10-14 17:29:45 EDT 
 m31001| 2013-10-14T21:28:10.318+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state SECONDARY
2013-10-14 17:29:48 EDT 
 m31000| 2013-10-14T21:28:16.140+0000 [rsMgr] replSet info electSelf 0
2013-10-14 17:29:50 EDT 
 m31001| 2013-10-14T21:28:16.140+0000 [conn2] replSet received elect msg { replSetElect: 1, set: "jstests_initsync1", who: "ip-10-142-147-116:31000", whoid: 0, cfgver: 1, round: ObjectId('525c61f0410e63258956f1dd') }
2013-10-14 17:29:53 EDT 
 m31001| 2013-10-14T21:28:16.140+0000 [conn2] replSet RECOVERING
2013-10-14 17:29:55 EDT 
 m31001| 2013-10-14T21:28:16.140+0000 [conn2] replSet info voting yea for ip-10-142-147-116:31000 (0)
2013-10-14 17:29:57 EDT 
 m31000| 2013-10-14T21:28:16.141+0000 [rsMgr] replSet elect res: { vote: 1, round: ObjectId('525c61f0410e63258956f1dd'), ok: 1.0 }
2013-10-14 17:30:00 EDT 
 m31000| 2013-10-14T21:28:17.140+0000 [rsMgr] replSet PRIMARY
2013-10-14 17:30:02 EDT 
 m31000| 2013-10-14T21:28:18.140+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state RECOVERING
2013-10-14 17:30:05 EDT 
 m31001| 2013-10-14T21:28:18.319+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state PRIMARY
2013-10-14 17:30:07 EDT 
 m31001| 2013-10-14T21:28:24.332+0000 [rsSync] replSet initial sync pending
2013-10-14 17:30:09 EDT 
 m31001| 2013-10-14T21:28:24.332+0000 [rsSync] replSet syncing to: ip-10-142-147-116:31000
2013-10-14 17:30:12 EDT 
 m31000| 2013-10-14T21:28:24.333+0000 [initandlisten] connection accepted from 10.142.147.116:58322 #3 (3 connections now open)
2013-10-14 17:30:14 EDT 
 m31001| 2013-10-14T21:28:24.334+0000 [rsSync] build index on: local.replset.minvalid properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.replset.minvalid", "name" : "_id_" }
2013-10-14 17:30:17 EDT 
 m31001| 2013-10-14T21:28:24.334+0000 [rsSync] build index done.  scanned 0 total records. 0 secs
2013-10-14 17:30:19 EDT 
 m31001| 2013-10-14T21:28:24.335+0000 [rsSync] replSet initial sync drop all databases
2013-10-14 17:30:22 EDT 
 m31001| 2013-10-14T21:28:24.335+0000 [rsSync] dropAllDatabasesExceptLocal 1
2013-10-14 17:30:24 EDT 
 m31001| 2013-10-14T21:28:24.335+0000 [rsSync] replSet initial sync clone all databases
2013-10-14 17:30:27 EDT 
 m31001| 2013-10-14T21:28:24.335+0000 [rsSync] replSet initial sync cloning db: admin
2013-10-14 17:30:29 EDT 
 m31000| 2013-10-14T21:28:24.336+0000 [initandlisten] connection accepted from 10.142.147.116:58323 #4 (4 connections now open)
2013-10-14 17:30:31 EDT 
 m31001| 2013-10-14T21:28:24.336+0000 [rsSync] replSet initial sync data copy, starting syncup
2013-10-14 17:30:34 EDT 
 m31001| 2013-10-14T21:28:24.336+0000 [rsSync] oplog sync 1 of 3
2013-10-14 17:30:37 EDT 
 m31001| 2013-10-14T21:28:24.336+0000 [rsSync] oplog sync 2 of 3
2013-10-14 17:30:44 EDT 
 m31001| 2013-10-14T21:28:24.337+0000 [rsSync] replSet initial sync building indexes
2013-10-14 17:30:46 EDT 
 m31001| 2013-10-14T21:28:24.337+0000 [rsSync] replSet initial sync cloning indexes for : admin
2013-10-14 17:30:49 EDT 
 m31001| 2013-10-14T21:28:24.338+0000 [rsSync] oplog sync 3 of 3
2013-10-14 17:30:51 EDT 
 m31001| 2013-10-14T21:28:24.338+0000 [rsSync] replSet initial sync finishing up
2013-10-14 17:30:54 EDT 
 m31000| 2013-10-14T21:28:24.339+0000 [conn4] end connection 10.142.147.116:58323 (3 connections now open)
2013-10-14 17:30:56 EDT 
 m31001| 2013-10-14T21:28:24.371+0000 [rsSync] replSet set minValid=525c61e7:b
2013-10-14 17:30:58 EDT 
 m31001| 2013-10-14T21:28:24.374+0000 [rsSync] replSet initial sync done
2013-10-14 17:31:01 EDT 
 m31000| 2013-10-14T21:28:24.374+0000 [conn3] end connection 10.142.147.116:58322 (2 connections now open)
2013-10-14 17:31:03 EDT 
 m31001| 2013-10-14T21:28:25.319+0000 [rsBackgroundSync] replSet syncing to: ip-10-142-147-116:31000
2013-10-14 17:31:06 EDT 
 m31000| 2013-10-14T21:28:25.320+0000 [initandlisten] connection accepted from 10.142.147.116:58325 #5 (3 connections now open)
2013-10-14 17:31:09 EDT 
 m31001| 2013-10-14T21:28:25.321+0000 [rsBackgroundSync] replSet remoteOldestOp:    Oct 14 21:28:07 525c61e7:b
2013-10-14 17:31:11 EDT 
 m31001| 2013-10-14T21:28:25.321+0000 [rsBackgroundSync] replSet lastOpTimeFetched: Oct 14 21:28:07 525c61e7:b
2013-10-14 17:31:13 EDT 
 m31000| 2013-10-14T21:28:25.321+0000 [initandlisten] connection accepted from 10.142.147.116:58326 #6 (4 connections now open)
 m31001| 2013-10-14T21:28:25.375+0000 [rsSync] replSet SECONDARY
2. Insert some data
 m31000| 2013-10-14T21:28:25.471+0000 [conn1] opening db:  foo
 m31000| 2013-10-14T21:28:25.472+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/foo.ns, filling with zeroes...
 m31000| 2013-10-14T21:28:25.474+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/foo.ns, size: 16MB,  took 0.002 secs
 m31000| 2013-10-14T21:28:25.476+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/foo.0, filling with zeroes...
 m31000| 2013-10-14T21:28:25.479+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/foo.0, size: 16MB,  took 0.002 secs
 m31000| 2013-10-14T21:28:25.479+0000 [conn1] datafileheader::init initializing /data/db/jstests_initsync1-0/foo.0 n:0
 m31000| 2013-10-14T21:28:25.480+0000 [conn1] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31000| 2013-10-14T21:28:25.480+0000 [conn1] have free list for foo.$freelist
 m31000| 2013-10-14T21:28:25.482+0000 [conn1] build index done.  scanned 0 total records. 0.001 secs
 m31001| 2013-10-14T21:28:25.488+0000 [repl prefetch worker] _DEBUG ReadContext db wasn't open, will try to open foo.bar
 m31001| 2013-10-14T21:28:25.488+0000 [repl prefetch worker] opening db:  foo
 m31001| 2013-10-14T21:28:25.490+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/foo.ns, filling with zeroes...
 m31001| 2013-10-14T21:28:25.492+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/foo.ns, size: 16MB,  took 0.002 secs
 m31001| 2013-10-14T21:28:25.495+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/foo.0, filling with zeroes...
 m31001| 2013-10-14T21:28:25.498+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/foo.0, size: 16MB,  took 0.003 secs
 m31001| 2013-10-14T21:28:25.499+0000 [repl writer worker 1] datafileheader::init initializing /data/db/jstests_initsync1-1/foo.0 n:0
 m31001| 2013-10-14T21:28:25.500+0000 [repl writer worker 1] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31001| 2013-10-14T21:28:25.501+0000 [repl writer worker 1] have free list for foo.$freelist
 m31001| 2013-10-14T21:28:25.502+0000 [repl writer worker 1] build index done.  scanned 0 total records. 0.002 secs
 m31000| 2013-10-14T21:28:26.142+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state SECONDARY
 m31000| 2013-10-14T21:28:26.339+0000 [slaveTracking] build index on: local.slaves properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.slaves", "name" : "_id_" }
 m31000| 2013-10-14T21:28:26.339+0000 [slaveTracking] build index done.  scanned 0 total records. 0 secs
total in foo: 10000
4. Make sure synced
ReplSetTest awaitReplication: starting: timestamp for primary, ip-10-142-147-116:31000, is Timestamp(1381786107, 3677)
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381786107, 3677)
ReplSetTest awaitReplication: checking secondary #1: ip-10-142-147-116:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-142-147-116:31001, is Timestamp(1381786107, 1478) but latest is Timestamp(1381786107, 3677)
ReplSetTest awaitReplication: last oplog entry (of 7802) for secondary #1, ip-10-142-147-116:31001, is {  "ts" : Timestamp(1381786107, 1478),  "h" : NumberLong("-3356782617010002881"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c61fa7a1d154a29acb352"),  "date" : ISODate("2013-10-14T21:28:26.510Z"),  "x" : 7800,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-142-147-116:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381786107, 3677)
ReplSetTest awaitReplication: checking secondary #1: ip-10-142-147-116:31001
ReplSetTest awaitReplication: secondary #1, ip-10-142-147-116:31001, is synced
ReplSetTest awaitReplication: finished: all 1 secondaries synced at timestamp Timestamp(1381786107, 3677)
5. Freeze #2
 m31001| 2013-10-14T21:28:28.726+0000 [conn1] replSet info 'freezing' for 999999 seconds
6. Bring up #3
2013-10-14T21:28:28.734+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --port 31002 --dbpath /data/db/jstests_initsync1 --noprealloc --smallfiles --oplogSize 2 --nohttpinterface --replSet jstests_initsync1 --setParameter enableTestCommands=1
2013-10-14T21:28:28.735+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31002| note: noprealloc may hurt performance in many applications
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] MongoDB starting : pid=9408 port=31002 dbpath=/data/db/jstests_initsync1 64-bit host=ip-10-142-147-116
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] _DEBUG build (which is slower)
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] 
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
2013-10-14 17:31:16 EDT 
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] **       Not recommended for production.
2013-10-14 17:31:18 EDT 
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] 
2013-10-14 17:31:20 EDT 
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] db version v2.5.3-pre-
2013-10-14 17:31:23 EDT 
 m31002| 2013-10-14T21:28:28.787+0000 [initandlisten] git version: 9fcdfeb195c42e91a49e37212a65fea624d71c13
2013-10-14 17:31:26 EDT 
 m31002| 2013-10-14T21:28:28.788+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
2013-10-14 17:31:28 EDT 
 m31002| 2013-10-14T21:28:28.788+0000 [initandlisten] allocator: tcmalloc
2013-10-14 17:31:31 EDT 
 m31002| 2013-10-14T21:28:28.788+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1", nohttpinterface: true, noprealloc: true, oplogSize: 2, port: 31002, replSet: "jstests_initsync1", setParameter: [ "enableTestCommands=1" ], smallfiles: true }
2013-10-14 17:31:33 EDT 
 m31002| 2013-10-14T21:28:28.792+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
2013-10-14 17:31:35 EDT 
 m31002| 2013-10-14T21:28:28.792+0000 [initandlisten] opening db:  admin
2013-10-14 17:31:38 EDT 
 m31002| 2013-10-14T21:28:28.793+0000 [initandlisten] 
2013-10-14 17:31:40 EDT 
 m31002| debug have W lock but w would suffice for command create
2013-10-14 17:31:43 EDT 
 m31002| 2013-10-14T21:28:28.794+0000 [initandlisten] opening db:  local
2013-10-14 17:31:45 EDT 
 m31002| 2013-10-14T21:28:28.794+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/local.ns, filling with zeroes...
2013-10-14 17:31:47 EDT 
 m31002| 2013-10-14T21:28:28.794+0000 [FileAllocator] creating directory /data/db/jstests_initsync1/_tmp
2013-10-14 17:31:50 EDT 
 m31002| 2013-10-14T21:28:28.799+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/local.ns, size: 16MB,  took 0.003 secs
2013-10-14 17:31:52 EDT 
 m31002| 2013-10-14T21:28:28.801+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/local.0, filling with zeroes...
2013-10-14 17:31:55 EDT 
 m31002| 2013-10-14T21:28:28.804+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/local.0, size: 16MB,  took 0.002 secs
2013-10-14 17:31:57 EDT 
 m31002| 2013-10-14T21:28:28.804+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1/local.0 n:0
2013-10-14 17:31:59 EDT 
 m31002| 2013-10-14T21:28:28.804+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
2013-10-14 17:32:02 EDT 
 m31002| 2013-10-14T21:28:28.804+0000 [initandlisten] waiting for connections on port 31002
2013-10-14 17:32:04 EDT 
 m31002| 2013-10-14T21:28:28.807+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
2013-10-14 17:32:07 EDT 
 m31002| 2013-10-14T21:28:28.807+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
2013-10-14 17:32:09 EDT 
 m31002| 2013-10-14T21:28:28.936+0000 [initandlisten] connection accepted from 127.0.0.1:42253 #1 (1 connection now open)
2013-10-14 17:32:11 EDT 
 m31000| 2013-10-14T21:28:28.938+0000 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
2013-10-14 17:32:14 EDT 
 m31002| 2013-10-14T21:28:28.939+0000 [initandlisten] connection accepted from 10.142.147.116:59093 #2 (2 connections now open)
2013-10-14 17:32:16 EDT 
 m31000| 2013-10-14T21:28:28.939+0000 [conn1] replSet replSetReconfig [2]
2013-10-14 17:32:19 EDT 
 m31000| 2013-10-14T21:28:28.939+0000 [conn1] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-142-147-116:31000" }, { _id: 1, host: "ip-10-142-147-116:31001" }, { _id: 2, host: "ip-10-142-147-116:31002" } ] }
2013-10-14 17:32:21 EDT 
 m31000| 2013-10-14T21:28:28.940+0000 [conn1] replSet saveConfigLocally done
2013-10-14 17:32:23 EDT 
 m31000| 2013-10-14T21:28:28.940+0000 [conn1] replSet info : additive change to configuration
2013-10-14 17:32:26 EDT 
 m31000| 2013-10-14T21:28:28.940+0000 [conn1] starting rsHealthPoll for ip-10-142-147-116:31002
2013-10-14 17:32:28 EDT 
 m31000| 2013-10-14T21:28:28.940+0000 [conn1] replSet replSetReconfig new config saved locally
2013-10-14 17:32:30 EDT 
 m31000| 2013-10-14T21:28:28.941+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is up
2013-10-14 17:32:33 EDT 
 m31002| 2013-10-14T21:28:28.943+0000 [conn1] _DEBUG ReadContext db wasn't open, will try to open foo.bar
2013-10-14 17:32:35 EDT 
 m31002| 2013-10-14T21:28:28.943+0000 [conn1] opening db:  foo
2013-10-14 17:32:37 EDT 
{
2013-10-14 17:32:40 EDT 
    "_id" : "jstests_initsync1",
2013-10-14 17:32:42 EDT 
    "version" : 1,
2013-10-14 17:32:44 EDT 
    "members" : [
2013-10-14 17:32:47 EDT 
        {
2013-10-14 17:32:50 EDT 
            "_id" : 0,
2013-10-14 17:32:52 EDT 
            "host" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-142-147-116:31001"
        }
    ]
}
null
waiting 3
 m31002| 2013-10-14T21:28:29.807+0000 [rsStart] trying to contact ip-10-142-147-116:31000
 m31000| 2013-10-14T21:28:29.809+0000 [initandlisten] connection accepted from 10.142.147.116:58333 #7 (5 connections now open)
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] replSet load config ok from ip-10-142-147-116:31000
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] replSet I am ip-10-142-147-116:31002
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31001
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31000
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] replSet got config version 2 from a remote, saving locally
 m31002| 2013-10-14T21:28:29.810+0000 [rsStart] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-142-147-116:31000" }, { _id: 1, host: "ip-10-142-147-116:31001" }, { _id: 2, host: "ip-10-142-147-116:31002" } ] }
 m31002| 2013-10-14T21:28:29.811+0000 [rsStart] replSet saveConfigLocally done
 m31002| 2013-10-14T21:28:29.811+0000 [rsStart] replSet STARTUP2
 m31002| 2013-10-14T21:28:29.812+0000 [rsSync] ******
 m31002| 2013-10-14T21:28:29.812+0000 [rsSync] creating replication oplog of size: 2MB...
 m31002| 2013-10-14T21:28:29.876+0000 [rsSync] ******
 m31002| 2013-10-14T21:28:29.876+0000 [rsSync] replSet initial sync pending
 m31002| 2013-10-14T21:28:29.876+0000 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
 m31002| 2013-10-14T21:28:29.878+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
 m31002| 2013-10-14T21:28:29.879+0000 [rsSyncNotifier] have free list for local.$freelist
 m31002| 2013-10-14T21:28:29.880+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
{
    "_id" : "jstests_initsync1",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-142-147-116:31001"
        }
    ]
}
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-142-147-116:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-142-147-116:31002"
        }
    ]
}
 m31001| 2013-10-14T21:28:30.321+0000 [rsMgr] replset msgReceivedNewConfig version: version: 2
 m31001| 2013-10-14T21:28:30.321+0000 [rsMgr] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-142-147-116:31000" }, { _id: 1, host: "ip-10-142-147-116:31001" }, { _id: 2, host: "ip-10-142-147-116:31002" } ] }
 m31001| 2013-10-14T21:28:30.322+0000 [rsMgr] replSet saveConfigLocally done
 m31001| 2013-10-14T21:28:30.322+0000 [rsMgr] replSet info : additive change to configuration
 m31001| 2013-10-14T21:28:30.322+0000 [rsMgr] starting rsHealthPoll for ip-10-142-147-116:31002
 m31001| 2013-10-14T21:28:30.322+0000 [rsMgr] replSet replSetReconfig new config saved locally
 m31002| 2013-10-14T21:28:30.323+0000 [initandlisten] connection accepted from 10.142.147.116:59096 #3 (3 connections now open)
 m31001| 2013-10-14T21:28:30.323+0000 [rsHealthPoll] replset info ip-10-142-147-116:31002 thinks that we are down
 m31001| 2013-10-14T21:28:30.323+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is up
 m31001| 2013-10-14T21:28:30.323+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state STARTUP2
 m31000| 2013-10-14T21:28:30.942+0000 [rsHealthPoll] replset info ip-10-142-147-116:31002 thinks that we are down
 m31000| 2013-10-14T21:28:30.942+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state STARTUP2
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-142-147-116:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-142-147-116:31002"
        }
    ]
}
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-142-147-116:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-142-147-116:31002"
        }
    ]
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:30Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : -1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:30Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : -1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:30Z"),
            "pingMs" : 0
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 2,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
 m31002| 2013-10-14T21:28:31.811+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is up
 m31002| 2013-10-14T21:28:31.811+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state PRIMARY
 m31001| 2013-10-14T21:28:31.811+0000 [initandlisten] connection accepted from 10.142.147.116:36919 #3 (3 connections now open)
 m31002| 2013-10-14T21:28:31.811+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is up
 m31002| 2013-10-14T21:28:31.811+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state SECONDARY
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:31Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 0,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:31Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:30Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 0,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:31Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:30Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 3,
            "optime" : Timestamp(0, 0),
2013-10-14 17:32:55 EDT 
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
2013-10-14 17:32:57 EDT 
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
2013-10-14 17:32:59 EDT 
            "self" : true
2013-10-14 17:33:02 EDT 
        }
2013-10-14 17:33:04 EDT 
    ],
2013-10-14 17:33:06 EDT 
    "ok" : 1
2013-10-14 17:33:09 EDT 
}
2013-10-14 17:33:11 EDT 
{
2013-10-14 17:33:13 EDT 
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:32Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 1,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:31Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:32Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 1,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:31Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:32Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 4,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:33Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 2,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:33Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:32Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 2,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:33Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:32Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 5,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
function () {
    var status = admin_s2.runCommand({replSetGetStatus:1});
    printjson(status);
    return status.members &&
      (status.members[2].state == 3 || status.members[2].state == 2);
  }
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:34Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 3,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:33Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:34Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 3,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:33Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:34Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 6,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:35Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 4,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:35Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:34Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 4,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:35Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:34Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 7,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31001| 2013-10-14T21:28:36.144+0000 [conn2] end connection 10.142.147.116:36876 (2 connections now open)
 m31001| 2013-10-14T21:28:36.144+0000 [initandlisten] connection accepted from 10.142.147.116:36923 #4 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:36Z"),
    "myState" : 5,
2013-10-14 17:33:16 EDT 
    "members" : [
2013-10-14 17:33:18 EDT 
        {
2013-10-14 17:33:20 EDT 
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 5,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:35Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:36Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 5,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:35Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:36Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 8,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:37Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 6,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:37Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:36Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 6,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:37Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:36Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 9,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31000| 2013-10-14T21:28:38.323+0000 [conn2] end connection 10.142.147.116:58309 (4 connections now open)
 m31000| 2013-10-14T21:28:38.323+0000 [initandlisten] connection accepted from 10.142.147.116:58344 #8 (5 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:38Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 7,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:37Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:38Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 7,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:37Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:38Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 10,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:39Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 8,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:39Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:38Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 8,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:39Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:38Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 11,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:40Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 9,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:39Z"),
2013-10-14 17:33:23 EDT 
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:40Z"),
2013-10-14 17:33:25 EDT 
            "pingMs" : 0
2013-10-14 17:33:27 EDT 
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 9,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:39Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:40Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 12,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:42Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 11,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:41Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:40Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 11,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:41Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:40Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 14,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-14T21:28:42.944+0000 [conn2] end connection 10.142.147.116:59093 (2 connections now open)
 m31002| 2013-10-14T21:28:42.945+0000 [initandlisten] connection accepted from 10.142.147.116:59111 #4 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:43Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 12,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:41Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:42Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 12,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:41Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:42Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 15,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:44Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 13,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:43Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:42Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 13,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:43Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:42Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 16,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-14T21:28:44.326+0000 [conn3] end connection 10.142.147.116:59096 (2 connections now open)
 m31002| 2013-10-14T21:28:44.326+0000 [initandlisten] connection accepted from 10.142.147.116:59113 #5 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:45Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 14,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:43Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:44Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
2013-10-14 17:33:30 EDT 
            "uptime" : 14,
2013-10-14 17:33:33 EDT 
            "optime" : Timestamp(1381786108, 1),
2013-10-14 17:33:35 EDT 
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
2013-10-14 17:33:37 EDT 
            "lastHeartbeat" : ISODate("2013-10-14T21:28:43Z"),
2013-10-14 17:33:40 EDT 
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:44Z"),
2013-10-14 17:33:42 EDT 
            "pingMs" : 0,
2013-10-14 17:33:45 EDT 
            "syncingTo" : "ip-10-142-147-116:31000"
2013-10-14 17:33:47 EDT 
        },
2013-10-14 17:33:50 EDT 
        {
2013-10-14 17:33:52 EDT 
            "_id" : 2,
2013-10-14 17:33:54 EDT 
            "name" : "ip-10-142-147-116:31002",
2013-10-14 17:33:57 EDT 
            "health" : 1,
2013-10-14 17:34:00 EDT 
            "state" : 5,
2013-10-14 17:34:02 EDT 
            "stateStr" : "STARTUP2",
2013-10-14 17:34:05 EDT 
            "uptime" : 17,
2013-10-14 17:34:12 EDT 
            "optime" : Timestamp(0, 0),
2013-10-14 17:34:15 EDT 
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
2013-10-14 17:34:17 EDT 
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
2013-10-14 17:34:20 EDT 
            "self" : true
2013-10-14 17:34:22 EDT 
        }
2013-10-14 17:34:25 EDT 
    ],
2013-10-14 17:34:27 EDT 
    "ok" : 1
2013-10-14 17:34:30 EDT 
}
2013-10-14 17:34:32 EDT 
 m31000| 2013-10-14T21:28:45.813+0000 [conn7] end connection 10.142.147.116:58333 (4 connections now open)
2013-10-14 17:34:34 EDT 
 m31000| 2013-10-14T21:28:45.814+0000 [initandlisten] connection accepted from 10.142.147.116:58353 #9 (5 connections now open)
2013-10-14 17:34:37 EDT 
 m31002| 2013-10-14T21:28:45.877+0000 [rsSync] replSet initial sync pending
2013-10-14 17:34:40 EDT 
 m31002| 2013-10-14T21:28:45.877+0000 [rsSync] replSet syncing to: ip-10-142-147-116:31000
2013-10-14 17:34:42 EDT 
 m31000| 2013-10-14T21:28:45.877+0000 [initandlisten] connection accepted from 10.142.147.116:58354 #10 (6 connections now open)
2013-10-14 17:34:44 EDT 
 m31002| 2013-10-14T21:28:45.878+0000 [rsSync] build index on: local.replset.minvalid properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.replset.minvalid", "name" : "_id_" }
2013-10-14 17:34:47 EDT 
 m31002| 2013-10-14T21:28:45.878+0000 [rsSync] build index done.  scanned 0 total records. 0 secs
2013-10-14 17:34:49 EDT 
 m31002| 2013-10-14T21:28:45.879+0000 [rsSync] replSet initial sync drop all databases
2013-10-14 17:34:52 EDT 
 m31002| 2013-10-14T21:28:45.879+0000 [rsSync] dropAllDatabasesExceptLocal 1
2013-10-14 17:34:54 EDT 
 m31002| 2013-10-14T21:28:45.879+0000 [rsSync] replSet initial sync clone all databases
2013-10-14 17:34:57 EDT 
 m31002| 2013-10-14T21:28:45.879+0000 [rsSync] replSet initial sync cloning db: foo
2013-10-14 17:34:59 EDT 
 m31000| 2013-10-14T21:28:45.879+0000 [initandlisten] connection accepted from 10.142.147.116:58355 #11 (7 connections now open)
2013-10-14 17:35:02 EDT 
 m31002| 2013-10-14T21:28:45.880+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/foo.ns, filling with zeroes...
2013-10-14 17:35:04 EDT 
 m31002| 2013-10-14T21:28:45.939+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/foo.ns, size: 16MB,  took 0.058 secs
2013-10-14 17:35:07 EDT 
 m31002| 2013-10-14T21:28:45.941+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/foo.0, filling with zeroes...
2013-10-14 17:35:09 EDT 
 m31002| 2013-10-14T21:28:45.943+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/foo.0, size: 16MB,  took 0.002 secs
2013-10-14 17:35:12 EDT 
 m31002| 2013-10-14T21:28:45.943+0000 [rsSync] datafileheader::init initializing /data/db/jstests_initsync1/foo.0 n:0
2013-10-14 17:35:15 EDT 
 m31000| 2013-10-14T21:28:45.944+0000 [conn11] exhaust=true sending more
2013-10-14 17:35:17 EDT 
{
2013-10-14 17:35:19 EDT 
    "set" : "jstests_initsync1",
2013-10-14 17:35:22 EDT 
    "date" : ISODate("2013-10-14T21:28:46Z"),
2013-10-14 17:35:24 EDT 
    "myState" : 5,
2013-10-14 17:35:27 EDT 
    "members" : [
2013-10-14 17:35:30 EDT 
        {
2013-10-14 17:35:32 EDT 
            "_id" : 0,
2013-10-14 17:35:34 EDT 
            "name" : "ip-10-142-147-116:31000",
2013-10-14 17:35:37 EDT 
            "health" : 1,
2013-10-14 17:35:39 EDT 
            "state" : 1,
2013-10-14 17:35:41 EDT 
            "stateStr" : "PRIMARY",
2013-10-14 17:35:44 EDT 
            "uptime" : 15,
2013-10-14 17:35:47 EDT 
            "optime" : Timestamp(1381786108, 1),
2013-10-14 17:35:49 EDT 
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
2013-10-14 17:35:57 EDT 
            "lastHeartbeat" : ISODate("2013-10-14T21:28:45Z"),
2013-10-14 17:35:59 EDT 
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:44Z"),
2013-10-14 17:36:02 EDT 
            "pingMs" : 0
2013-10-14 17:36:04 EDT 
        },
2013-10-14 17:36:07 EDT 
        {
2013-10-14 17:36:09 EDT 
            "_id" : 1,
2013-10-14 17:36:12 EDT 
            "name" : "ip-10-142-147-116:31001",
2013-10-14 17:36:14 EDT 
            "health" : 1,
2013-10-14 17:36:16 EDT 
            "state" : 2,
2013-10-14 17:36:19 EDT 
            "stateStr" : "SECONDARY",
2013-10-14 17:36:22 EDT 
            "uptime" : 15,
2013-10-14 17:36:24 EDT 
            "optime" : Timestamp(1381786108, 1),
2013-10-14 17:36:26 EDT 
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
2013-10-14 17:36:29 EDT 
            "lastHeartbeat" : ISODate("2013-10-14T21:28:45Z"),
2013-10-14 17:36:31 EDT 
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:44Z"),
2013-10-14 17:36:34 EDT 
            "pingMs" : 0,
2013-10-14 17:36:36 EDT 
            "syncingTo" : "ip-10-142-147-116:31000"
2013-10-14 17:36:39 EDT 
        },
2013-10-14 17:36:41 EDT 
        {
2013-10-14 17:36:44 EDT 
            "_id" : 2,
2013-10-14 17:36:46 EDT 
            "name" : "ip-10-142-147-116:31002",
2013-10-14 17:36:49 EDT 
            "health" : 1,
2013-10-14 17:36:51 EDT 
            "state" : 5,
2013-10-14 17:36:54 EDT 
            "stateStr" : "STARTUP2",
2013-10-14 17:36:56 EDT 
            "uptime" : 18,
2013-10-14 17:36:58 EDT 
            "optime" : Timestamp(0, 0),
2013-10-14 17:37:01 EDT 
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
2013-10-14 17:37:03 EDT 
            "errmsg" : "initial sync cloning db: foo",
2013-10-14 17:37:05 EDT 
            "self" : true
2013-10-14 17:37:08 EDT 
        }
2013-10-14 17:37:10 EDT 
    ],
    "ok" : 1
}
 m31002| 2013-10-14T21:28:46.177+0000 [rsSync] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31002| 2013-10-14T21:28:46.177+0000 [rsSync] have free list for foo.$freelist
 m31002| 2013-10-14T21:28:46.284+0000 [rsSync]   fastBuildIndex dupsToDrop:0
 m31002| 2013-10-14T21:28:46.286+0000 [rsSync] build index done.  scanned 10000 total records. 0.108 secs
 m31002| 2013-10-14T21:28:46.286+0000 [rsSync] replSet initial sync cloning db: admin
 m31002| 2013-10-14T21:28:46.286+0000 [rsSync] replSet initial sync data copy, starting syncup
 m31002| 2013-10-14T21:28:46.286+0000 [rsSync] oplog sync 1 of 3
 m31002| 2013-10-14T21:28:46.286+0000 [rsSync] oplog sync 2 of 3
 m31002| 2013-10-14T21:28:46.287+0000 [rsSync] replSet initial sync building indexes
 m31002| 2013-10-14T21:28:46.287+0000 [rsSync] replSet initial sync cloning indexes for : foo
 m31002| 2013-10-14T21:28:46.287+0000 [rsSync] replSet initial sync cloning indexes for : admin
 m31002| 2013-10-14T21:28:46.287+0000 [rsSync] oplog sync 3 of 3
 m31000| 2013-10-14T21:28:46.287+0000 [conn11] end connection 10.142.147.116:58355 (6 connections now open)
 m31002| 2013-10-14T21:28:46.288+0000 [rsSync] replSet initial sync finishing up
 m31002| 2013-10-14T21:28:46.294+0000 [rsSync] replSet set minValid=525c61fc:1
 m31002| 2013-10-14T21:28:46.297+0000 [rsSync] replSet RECOVERING
 m31002| 2013-10-14T21:28:46.297+0000 [rsSync] replSet initial sync done
 m31000| 2013-10-14T21:28:46.297+0000 [conn10] end connection 10.142.147.116:58354 (5 connections now open)
 m31001| 2013-10-14T21:28:46.327+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state RECOVERING
 m31002| 2013-10-14T21:28:46.813+0000 [rsBackgroundSync] replSet syncing to: ip-10-142-147-116:31000
 m31000| 2013-10-14T21:28:46.814+0000 [initandlisten] connection accepted from 10.142.147.116:58357 #12 (6 connections now open)
 m31002| 2013-10-14T21:28:46.814+0000 [rsBackgroundSync] replSet remoteOldestOp:    Oct 14 21:28:07 525c61e7:b
 m31002| 2013-10-14T21:28:46.814+0000 [rsBackgroundSync] replSet lastOpTimeFetched: Oct 14 21:28:28 525c61fc:1
 m31000| 2013-10-14T21:28:46.815+0000 [initandlisten] connection accepted from 10.142.147.116:58358 #13 (7 connections now open)
 m31000| 2013-10-14T21:28:46.946+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state RECOVERING
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:47Z"),
    "myState" : 3,
    "syncingTo" : "ip-10-142-147-116:31000",
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 16,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:45Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:46Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 16,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:45Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:46Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 3,
            "stateStr" : "RECOVERING",
            "uptime" : 19,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "errmsg" : "syncing to: ip-10-142-147-116:31000",
            "self" : true
        }
    ],
    "ok" : 1
}
7. Kill #2 in the middle of syncing
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
ReplSetTest stop *** Shutting down mongod in port 31001 ***
 m31002| 2013-10-14T21:28:47.027+0000 [conn5] end connection 10.142.147.116:59113 (2 connections now open)
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] now exiting
 m31001| dbexit: 2013-10-14T21:28:47.027+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] closing listening socket: 11
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] closing listening socket: 12
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] closing listening socket: 15
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31001| 2013-10-14T21:28:47.027+0000 [signalProcessingThread] shutdown: closing all files...
 m31001| 2013-10-14T21:28:47.027+0000 [conn4] end connection 10.142.147.116:36923 (2 connections now open)
 m31001| 2013-10-14T21:28:47.028+0000 [conn3] end connection 10.142.147.116:36919 (2 connections now open)
 m31000| 2013-10-14T21:28:47.028+0000 [conn8] end connection 10.142.147.116:58344 (6 connections now open)
 m31001| 2013-10-14T21:28:47.028+0000 [conn1] end connection 127.0.0.1:59039 (2 connections now open)
 m31000| 2013-10-14T21:28:47.028+0000 [conn6] end connection 10.142.147.116:58326 (5 connections now open)
 m31001| 2013-10-14T21:28:47.030+0000 [signalProcessingThread] closeAllFiles() finished
 m31001| 2013-10-14T21:28:47.030+0000 [signalProcessingThread] shutdown: removing fs lock...
 m31001| 2013-10-14T21:28:47.030+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: ip-10-142-147-116:31000
 m31002| 2013-10-14T21:28:47.297+0000 [rsSync] replSet SECONDARY
 m31002| 2013-10-14T21:28:47.814+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:28:47.814+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31001 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31002" }
 m31002| 2013-10-14T21:28:47.814+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:47.816+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:47.816+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:47.816+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:47.816+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31001 is down (or slow to respond): 
 m31002| 2013-10-14T21:28:47.816+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state DOWN
2013-10-14T21:28:48.027+0000 shell: stopped mongo program on port 31001
8. Eventually it should become a secondary
if initial sync has started, this will cause it to fail and sleep for 5 minutes
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:28:48Z"),
    "myState" : 2,
    "syncingTo" : "ip-10-142-147-116:31000",
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 17,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:47Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:46Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:28:47Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-14T21:28:46Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-142-147-116:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 20,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "self" : true
        }
    ],
    "ok" : 1
}
9. Bring #2 back up
ReplSetTest n is : 1
 m31000| 2013-10-14T21:28:48.147+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:28:48.147+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31001 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31000" }
 m31000| 2013-10-14T21:28:48.147+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:28:48.148+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:28:48.148+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:28:48.149+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:28:48.149+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31001 is down (or slow to respond): 
 m31000| 2013-10-14T21:28:48.149+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state DOWN
 m31000| 2013-10-14T21:28:48.946+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state SECONDARY
 m31000| 2013-10-14T21:28:49.394+0000 [conn5] end connection 10.142.147.116:58325 (4 connections now open)
 m31000| 2013-10-14T21:28:50.149+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:28:50.149+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:28:50.151+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:28:50.151+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
 m31000| 2013-10-14T21:28:50.151+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:28:50.151+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:28:50.156+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:28:50.156+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:28:50.156+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:28:52.156+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:12 EDT 
 m31000| 2013-10-14T21:28:52.156+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:15 EDT 
 m31000| 2013-10-14T21:28:52.165+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:37:17 EDT 
 m31000| 2013-10-14T21:28:52.165+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
2013-10-14 17:37:19 EDT 
 m31000| 2013-10-14T21:28:52.165+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:22 EDT 
 m31000| 2013-10-14T21:28:52.165+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:24 EDT 
 m31000| 2013-10-14T21:28:52.181+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:37:27 EDT 
 m31000| 2013-10-14T21:28:52.181+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:30 EDT 
 m31000| 2013-10-14T21:28:52.181+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:32 EDT 
 m31000| 2013-10-14T21:28:54.182+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:34 EDT 
 m31000| 2013-10-14T21:28:54.182+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:37 EDT 
 m31000| 2013-10-14T21:28:54.214+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:37:39 EDT 
 m31000| 2013-10-14T21:28:54.215+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
2013-10-14 17:37:41 EDT 
 m31000| 2013-10-14T21:28:54.215+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:44 EDT 
 m31000| 2013-10-14T21:28:54.215+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:46 EDT 
 m31000| 2013-10-14T21:28:54.279+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:37:49 EDT 
 m31000| 2013-10-14T21:28:54.279+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:51 EDT 
 m31000| 2013-10-14T21:28:54.279+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:54 EDT 
 m31000| 2013-10-14T21:28:56.280+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:37:56 EDT 
 m31000| 2013-10-14T21:28:56.280+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:37:58 EDT 
 m31000| 2013-10-14T21:28:56.408+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:38:01 EDT 
 m31000| 2013-10-14T21:28:56.409+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
2013-10-14 17:38:03 EDT 
 m31000| 2013-10-14T21:28:56.409+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:05 EDT 
 m31000| 2013-10-14T21:28:56.409+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:38:08 EDT 
 m31000| 2013-10-14T21:28:56.665+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:38:11 EDT 
 m31000| 2013-10-14T21:28:56.665+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:13 EDT 
 m31000| 2013-10-14T21:28:56.665+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:38:15 EDT 
 m31000| 2013-10-14T21:28:58.666+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:18 EDT 
 m31000| 2013-10-14T21:28:58.666+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:38:20 EDT 
 m31000| 2013-10-14T21:28:59.178+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:28:59.178+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:28:59.178+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:28:59.178+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:00.179+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:00.179+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:00.179+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:02.180+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:02.180+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:02.181+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31001,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : true,
    "pathOpts" : {
        "node" : 1,
        "set" : "jstests_initsync1"
    }
}
 m31000| 2013-10-14T21:29:02.181+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:02.181+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:02.181+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:02.184+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:02.184+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:02.184+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:04.184+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:04.184+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:49.816+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:49.816+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:04.189+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:04.189+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
ReplSetTest (Re)Starting....
 m31000| 2013-10-14T21:29:04.189+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:04.189+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:04.198+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:04.198+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:04.198+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:06.198+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:06.198+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:06.216+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:06.216+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:06.216+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:06.216+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:06.248+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:06.248+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:06.248+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:07.115+0000 [clientcursormon] mem (MB) res:71 virt:744
 m31000| 2013-10-14T21:29:07.115+0000 [clientcursormon]  mapped:128
 m31000| 2013-10-14T21:29:07.115+0000 [clientcursormon]  connections:4
 m31000| 2013-10-14T21:29:07.115+0000 [clientcursormon]  replication threads:32
 m31000| 2013-10-14T21:29:08.249+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:08.249+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:08.249+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.249+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:08.250+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.250+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:08.251+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.251+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.252+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:08.254+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.255+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:08.255+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:08.255+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:08.255+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.256+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:49.819+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14T21:32:28.613+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-1 --setParameter enableTestCommands=1
 m31002| 2013-10-14T21:28:49.819+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:49.819+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:49.819+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:49.823+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:49.823+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:49.823+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:51.824+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:51.824+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:51.832+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:51.832+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:51.832+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:51.832+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:51.849+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:51.849+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:51.849+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:53.849+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:53.849+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:53.882+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:53.882+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:53.882+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:53.882+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:53.947+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:53.947+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:53.947+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:55.947+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:55.947+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:56.076+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:56.076+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:56.076+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:56.076+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:56.333+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:56.333+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:56.333+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:58.333+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:58.333+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:58.846+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:58.846+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:28:58.846+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:58.846+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:28:59.847+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:28:59.847+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:28:59.847+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:01.847+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:01.847+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14T21:32:28.614+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.848+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.848+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:01.849+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.849+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:01.850+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.851+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:01.851+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.851+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:01.851+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:01.851+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:01.852+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.852+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:01.854+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.855+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:01.855+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:01.855+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:01.855+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.856+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:03.861+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:03.862+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.862+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:03.870+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:03.871+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
[
    connection to ip-10-142-147-116:31000,
    connection to ip-10-142-147-116:31001
]
 m31002| 2013-10-14T21:29:03.871+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:03.871+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:03.871+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:05.871+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.872+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:05.872+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:05.872+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:05.872+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.872+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:05.889+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:05.890+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.890+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:05.922+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.923+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:05.923+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:05.923+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:05.923+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:07.924+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:07.991+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:07.991+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:38:23 EDT 
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:25 EDT 
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:38:27 EDT 
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:07.992+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:08.121+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14T21:33:13.676+0000 m31002| 2013-10-14T21:29:08.121+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 DBClientCursor::init call() failed
 m31002| 2013-10-14T21:29:08.121+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:08.122+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:08.122+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:10.122+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.122+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:10.122+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:10.122+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:10.123+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.123+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:10.380+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:10.381+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.381+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:10.894+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.894+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:10.895+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:10.895+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:10.895+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:12.895+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:12.895+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:12.896+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:12.896+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:12.896+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:12.896+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:12.951+0000 [conn4] end connection 10.142.147.116:59111 (1 connection now open)
 m31002| 2013-10-14T21:29:12.951+0000 [initandlisten] connection accepted from 10.142.147.116:59225 #6 (2 connections now open)
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
Error: error doing query: failed
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:13.897+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:13.898+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:13.898+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
waiting 6
 m31002| 2013-10-14T21:29:14.899+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:14.900+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:14.900+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:14.900+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:14.900+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:16.900+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.900+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:16.900+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:16.900+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:16.901+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.901+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:16.902+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.902+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.903+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:16.906+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.907+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:16.907+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:16.907+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:16.907+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.908+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14T21:34:31.041+0000 trying reconnect to 127.0.0.1:31001
 m31002| 2013-10-14T21:29:18.913+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:18.914+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.914+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:18.922+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:18.923+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.659+0000 ** WARNING: --rest is specified without --httpinterface,
2013-10-14 17:38:34 EDT 
 m31002| 2013-10-14T21:29:18.923+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:38:37 EDT 
 m31002| 2013-10-14T21:29:18.923+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:39 EDT 
 m31002| 2013-10-14T21:29:18.923+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:38:41 EDT 
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:20.924+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.924+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:20.924+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:20.924+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:20.925+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.925+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:20.941+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.941+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14T21:37:10.369+0000 reconnect 127.0.0.1:31001 ok
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.261+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:10.262+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.262+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:10.270+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.271+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:10.271+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:10.271+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:10.271+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:12.271+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:12.271+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:12.272+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:12.272+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:12.272+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:12.272+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:12.289+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:20.942+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:20.975+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.976+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:20.976+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:20.976+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:20.976+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:22.977+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:22.977+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:22.977+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:22.977+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:22.977+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:22.978+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:23.042+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-14T21:37:10Z"),
    "myState" : 2,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-142-147-116:31000",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:37:03Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-142-147-116:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 282,
            "optime" : Timestamp(1381786108, 1),
            "optimeDate" : ISODate("2013-10-14T21:28:28Z"),
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "ip-10-142-147-116:31002",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2013-10-14T21:37:05Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-14T21:29:23.042+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:23.043+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:23.043+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:23.043+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:23.043+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:23.044+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.659+0000 **          enabling http interface
 m31001| note: noprealloc may hurt performance in many applications
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] MongoDB starting : pid=3410 port=31001 dbpath=/data/db/jstests_initsync1-1 64-bit host=ip-10-142-147-116
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] _DEBUG build (which is slower)
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] 
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] **       Not recommended for production.
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] 
10. Insert some stuff
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] db version v2.5.3-pre-
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] git version: 9fcdfeb195c42e91a49e37212a65fea624d71c13
 m31002| 2013-10-14T21:29:23.044+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:12.290+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] allocator: tcmalloc
 m31000| 2013-10-14T21:29:12.290+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:23.172+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.666+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31001| 2013-10-14T21:32:28.670+0000 [initandlisten] opening db:  local
 m31001| 2013-10-14T21:32:28.672+0000 [initandlisten] opening db:  foo
 m31000| 2013-10-14T21:29:12.322+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:23.173+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:23.173+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:12.323+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:23.173+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:12.323+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31001| 2013-10-14T21:32:28.674+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31001| 2013-10-14T21:32:28.674+0000 [initandlisten] opening db:  admin
 m31000| 2013-10-14T21:29:12.323+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:12.323+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:23.173+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31001| 2013-10-14T21:32:28.675+0000 [IndexRebuilder] _DEBUG ReadContext db wasn't open, will try to open local.system.namespaces
 m31001| 2013-10-14T21:32:28.675+0000 [IndexRebuilder] opening db:  local
 m31002| 2013-10-14T21:29:25.174+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:25.174+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31001| 2013-10-14T21:32:28.676+0000 [initandlisten] 
 m31002| 2013-10-14T21:29:25.174+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| debug have W lock but w would suffice for command create
 m31002| 2013-10-14T21:29:25.174+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.324+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14T21:37:10.382+0000 Socket say send() errno:32 Broken pipe 127.0.0.1:31000
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:14.389+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:14.390+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.390+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:14.519+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.519+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:14.519+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:14.519+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:14.519+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:15.820+0000 [conn9] end connection 10.142.147.116:58353 (3 connections now open)
 m31000| 2013-10-14T21:29:15.821+0000 [initandlisten] connection accepted from 10.142.147.116:58478 #14 (4 connections now open)
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:16.520+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:16.777+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:16.777+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:16.778+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:16.778+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:16.778+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:16.778+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:16.779+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:16.779+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:17.291+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:17.292+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:17.292+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:17.292+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:17.292+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| 2013-10-14T21:32:28.676+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
ReplSetTest Could not call ismaster on node 0: Error: socket exception [SEND_ERROR] for 127.0.0.1:31000
 m31002| 2013-10-14T21:29:25.175+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:25.175+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:25.431+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.676+0000 [initandlisten] waiting for connections on port 31001
 m31001| 2013-10-14T21:32:28.677+0000 [websvr] admin web console waiting for connections on port 32001
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.677+0000 [IndexRebuilder] _DEBUG ReadContext db wasn't open, will try to open foo.system.namespaces
 m31001| 2013-10-14T21:32:28.677+0000 [IndexRebuilder] opening db:  foo
 m31001| 2013-10-14T21:32:28.679+0000 [IndexRebuilder] assertion 17069 not master and slaveOk=false ns:foo.system.namespaces query:{}
 m31001| 2013-10-14T21:32:28.680+0000 [IndexRebuilder] warning: index rebuilding did not complete
 m31001| 2013-10-14T21:32:28.681+0000 [rsStart] replSet I am ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:38:44 EDT 
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:38:46 EDT 
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:38:48 EDT 
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:25.432+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:25.433+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:25.945+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:25.946+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:25.946+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:25.946+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:25.946+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:27.947+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:28.794+0000 [clientcursormon] mem (MB) res:57 virt:644
 m31002| 2013-10-14T21:29:28.794+0000 [clientcursormon]  mapped:64
 m31002| 2013-10-14T21:29:28.794+0000 [clientcursormon]  connections:2
 m31002| 2013-10-14T21:29:28.794+0000 [clientcursormon]  replication threads:32
 m31002| 2013-10-14T21:29:28.948+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:32:28.681+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31002
 m31001| 2013-10-14T21:32:28.681+0000 [rsStart] starting rsHealthPoll for ip-10-142-147-116:31000
assert.soon failed: function () {
      master = self.callIsMaster();
      return master;
    }, msg:Finding master
 m31001| 2013-10-14T21:32:28.681+0000 [rsStart] replSet STARTUP2
 m31001| 2013-10-14T21:32:28.682+0000 [rsSync] replSet SECONDARY
 m31001| 2013-10-14T21:32:38.682+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:32:38.682+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31000 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31001" }
 m31001| 2013-10-14T21:32:38.682+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31000 is down (or slow to respond): 
 m31001| 2013-10-14T21:32:38.682+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state DOWN
 m31001| 2013-10-14T21:32:38.682+0000 [rsMgr] replSet can't see a majority, will not try to elect self
 m31001| 2013-10-14T21:32:40.681+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:32:40.681+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31002 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31001" }
 m31001| 2013-10-14T21:32:40.681+0000 [rsHealthPoll] replset info ip-10-142-147-116:31002 heartbeat failed, retrying
 m31001| 2013-10-14T21:32:41.683+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:32:41.683+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31002 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31001" }
 m31001| 2013-10-14T21:32:41.683+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31002 is down (or slow to respond): 
 m31001| 2013-10-14T21:32:41.683+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state DOWN
 m31001| 2013-10-14T21:32:43.683+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:32:50.684+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:32:52.338+0000 [initandlisten] connection accepted from 127.0.0.1:60074 #1 (1 connection now open)
 m31001| 2013-10-14T21:32:53.685+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:32:58.684+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:33:02.686+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:05.687+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:13.685+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:33:14.688+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:17.689+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:26.690+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:28.676+0000 [clientcursormon] mem (MB) res:46 virt:714
 m31001| 2013-10-14T21:33:28.676+0000 [clientcursormon]  mapped:128
 m31001| 2013-10-14T21:33:28.676+0000 [clientcursormon]  connections:1
 m31001| 2013-10-14T21:33:28.676+0000 [clientcursormon]  replication threads:32
 m31001| 2013-10-14T21:33:28.686+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:33:29.691+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:38.692+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:41.693+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:43.687+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:33:50.694+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:53.695+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:33:58.689+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:34:02.695+0000 [rsHealthPoll] DBClientCursor::init call() failed
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:38:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:174:60)
    at ReplSetTest.getMaster (src/mongo/shell/replsettest.js:410:12)
    at /data/mci/git@github.commongodb/mongo.git/master/jstests/replsets/initial_sync1.js:118:19
 m31001| 2013-10-14T21:34:05.697+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:13.690+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:34:14.697+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:17.699+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:26.699+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:28.691+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:34:29.701+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:38.701+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:41.703+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:43.692+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:34:50.703+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:53.704+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:34:58.694+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:35:02.705+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:05.706+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:13.695+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:35:14.707+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:17.708+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:26.709+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:28.696+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:35:29.709+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:38.710+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:41.711+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:43.697+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:35:50.712+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:53.712+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:35:58.698+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:36:02.714+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:05.713+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:13.699+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:36:14.716+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:17.715+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:26.718+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:28.701+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:36:29.717+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:38.720+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:41.719+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:43.702+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:36:50.722+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:53.721+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:36:58.703+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:37:02.722+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:02.722+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 heartbeat failed, retrying
 m31001| 2013-10-14T21:37:03.724+0000 [rsHealthPoll] DBClientCursor::init call() failed
ReplSetTest getMaster failed: "assert.soon failed: function () {\n      master = self.callIsMaster();\n      return master;\n    }, msg:Finding master"
 m31001| 2013-10-14T21:37:05.723+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:10.369+0000 [initandlisten] connection accepted from 127.0.0.1:60345 #2 (2 connections now open)
 m31001| 2013-10-14T21:37:13.704+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:37:15.726+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:17.725+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:27.728+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:28.706+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:37:29.727+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:39.730+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:41.729+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:43.707+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:37:51.732+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:53.731+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:37:58.708+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:38:03.734+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:38:05.733+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:38:13.709+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:38:15.736+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:38:17.735+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:38:27.738+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:29:28.948+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:28.949+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:29.950+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:29.950+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:29.951+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:29.951+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:29.951+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:31.951+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.951+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:31.951+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:31.951+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:31.952+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.952+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:38:15)
    at ReplSetTest.getMaster (src/mongo/shell/replsettest.js:417:5)
    at /data/mci/git@github.commongodb/mongo.git/master/jstests/replsets/initial_sync1.js:118:19
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:31.954+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:31.955+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.955+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:31.957+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.958+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:31.958+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:31.958+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:31.958+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.959+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:33.964+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.964+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:33.964+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.964+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:33.964+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:33.965+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:33.965+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.965+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:33.973+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.974+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:33.974+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:33.974+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:33.974+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:35.975+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:35.975+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:35.975+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:35.975+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14T21:38:27.757+0000  m31002| 2013-10-14T21:29:35.976+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
assert.soon failed: function () {
      master = self.callIsMaster();
      return master;
    }, msg:Finding master at src/mongo/shell/replsettest.js:418
failed to load: /data/mci/git@github.commongodb/mongo.git/master/jstests/replsets/initial_sync1.js
 m31002| 2013-10-14T21:29:35.976+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:38:51 EDT 
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
2013-10-14 17:38:53 EDT 
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:38:55 EDT 
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:35.993+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:35.994+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:35.994+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:36.027+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:36.027+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:36.027+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:36.027+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:36.027+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:38.029+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:38:28.682+0000 [clientcursormon] mem (MB) res:47 virt:716
 m31001| 2013-10-14T21:38:28.682+0000 [clientcursormon]  mapped:128
 m31002| 2013-10-14T21:29:38.029+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:19.293+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31001| 2013-10-14T21:38:28.682+0000 [clientcursormon]  connections:2
 m31002| 2013-10-14T21:29:38.029+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:38.029+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:38.030+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:38:28.682+0000 [clientcursormon]  replication threads:32
 m31002| 2013-10-14T21:29:38.030+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:38.030+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:38.030+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:38.032+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:20.294+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:20.295+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:20.295+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:21.296+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:21.296+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:21.296+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:21.296+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:21.296+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:23.297+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.297+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:23.297+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:23.297+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:23.298+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.298+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:23.299+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.299+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:23.300+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.300+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:23.300+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:23.300+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:23.301+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.301+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:23.304+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.304+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:23.304+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:23.304+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:23.304+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.305+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:25.310+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.310+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:25.310+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.310+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:25.311+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:25.311+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:25.311+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:38.032+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:38.032+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.311+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31001| 2013-10-14T21:38:28.710+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:38:29.736+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:29:25.320+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.320+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:25.320+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:25.320+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:25.320+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31001| 2013-10-14T21:38:39.740+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:38.032+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:38.032+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:38.033+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:38.033+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:38.033+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:38.036+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:38.037+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:38.037+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:38.037+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:38.037+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:40.038+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.038+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:27.321+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:40.038+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:40.038+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:40.039+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.039+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:40.043+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.043+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.044+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:40.053+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.054+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:40.054+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:40.054+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:40.054+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:42.055+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.055+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:42.055+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:42.055+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:42.056+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.056+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:42.072+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.072+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.073+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:42.106+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.107+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:42.107+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:42.107+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:42.107+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:42.958+0000 [conn6] end connection 10.142.147.116:59225 (1 connection now open)
 m31002| 2013-10-14T21:29:42.959+0000 [initandlisten] connection accepted from 10.142.147.116:59424 #7 (2 connections now open)
 m31002| 2013-10-14T21:29:44.107+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.107+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:44.108+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:44.108+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:44.108+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.108+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:44.173+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:44.174+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.174+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:44.303+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.303+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:44.303+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:44.303+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:44.303+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:46.304+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:46.304+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:46.304+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:46.304+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:46.305+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:46.305+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:46.561+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:46.561+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:46.562+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:47.075+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:47.076+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:47.076+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:47.076+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:47.076+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:49.077+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:27.338+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:27.338+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:27.339+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:27.372+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:50.078+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:50.079+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:50.079+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:51.080+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:51.080+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:51.080+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:38:57 EDT 
 m31002| 2013-10-14T21:29:51.080+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:39:00 EDT 
 m31002| 2013-10-14T21:29:51.080+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:39:02 EDT 
 m31000| 2013-10-14T21:29:27.372+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:04 EDT 
 m31000| 2013-10-14T21:29:27.372+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:06 EDT 
 m31000| 2013-10-14T21:29:27.372+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:39:08 EDT 
 m31000| 2013-10-14T21:29:27.372+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:39:10 EDT 
 m31002| 2013-10-14T21:29:53.081+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.081+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.081+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:29.373+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.373+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:29.373+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:29.373+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:29.374+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.081+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:29.374+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:29.438+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.082+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.082+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.083+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.083+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:53.085+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.085+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.085+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:53.085+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:53.086+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.086+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.088+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.439+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:29.568+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.569+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:29.569+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:29.569+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:29.569+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:31.570+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:31.827+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:31.828+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:31.828+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:32.341+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:32.341+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:32.341+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:32.341+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:32.341+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:34.342+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:34.342+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:34.342+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:34.342+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:34.343+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:34.343+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:35.344+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:35.345+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:35.345+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:36.346+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:36.346+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:36.346+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:36.346+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:36.346+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.347+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:38.349+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.349+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.350+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:38.353+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.353+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:38.353+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:38.353+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:38.353+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:40.354+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.354+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:40.354+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:40.354+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:40.355+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.355+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:40.359+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.359+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.360+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:40.369+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.370+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:40.370+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:40.370+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:40.370+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.371+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:42.388+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:42.389+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.389+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:42.421+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.422+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:42.422+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:42.422+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:42.422+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:44.422+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.423+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.423+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:44.423+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:44.423+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.423+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.424+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.424+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.425+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.425+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.426+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.429+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.429+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:44.429+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:44.429+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:44.429+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:53.089+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:53.089+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:53.089+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:53.089+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.091+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:55.096+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:55.097+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.097+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:55.106+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:45.827+0000 [conn14] end connection 10.142.147.116:58478 (3 connections now open)
 m31000| 2013-10-14T21:29:45.828+0000 [initandlisten] connection accepted from 10.142.147.116:58681 #15 (4 connections now open)
 m31000| 2013-10-14T21:29:46.430+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.430+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:46.430+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:46.430+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:46.431+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.431+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:46.435+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.435+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.436+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:46.445+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.106+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:55.106+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:55.106+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:55.106+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:38:41.738+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.445+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:46.445+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:46.445+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:46.445+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:39:13 EDT 
 m31000| 2013-10-14T21:29:48.446+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:15 EDT 
 m31000| 2013-10-14T21:29:48.446+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:17 EDT 
 m31000| 2013-10-14T21:29:48.446+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:48.446+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:48.447+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.447+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:48.463+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.464+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:48.464+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.464+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:48.464+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:48.464+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:48.465+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.465+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:48.497+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.498+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:48.498+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:48.498+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:48.498+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.499+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:50.564+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.564+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.565+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:50.694+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.694+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:50.694+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31001| 2013-10-14T21:38:43.712+0000 [rsBackgroundSync] waiting for 6 pings from other members before syncing
 m31001| 2013-10-14T21:38:48.499+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-14T21:38:48.499+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 heartbeat failed, retrying
 m31001| 2013-10-14T21:38:48.501+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:38:48.501+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| 2013-10-14T21:38:48.501+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31001| 2013-10-14T21:38:48.502+0000 [initandlisten] connection accepted from 10.142.147.116:38291 #3 (3 connections now open)
 m31000| 2013-10-14T21:29:50.694+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:50.694+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:52.695+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:52.695+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:52.695+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:52.695+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:52.696+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:52.696+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31001| 2013-10-14T21:38:48.510+0000 [conn3] end connection 10.142.147.116:38291 (2 connections now open)
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:52.953+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:52.954+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:52.954+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:53.466+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:53.467+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:53.467+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:53.467+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:53.467+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:55.468+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:56.469+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:56.469+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:56.470+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:57.471+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:57.472+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:57.472+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:57.472+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:57.472+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.473+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:59.475+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.475+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:29:59.475+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.476+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:59.476+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:59.476+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:29:59.476+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.476+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:59.478+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.479+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:29:59.479+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:29:59.479+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:29:59.479+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:01.480+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.480+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:01.480+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:01.480+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:01.481+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.481+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:01.485+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.485+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.486+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:01.495+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.495+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:01.495+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:01.495+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:01.495+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:03.496+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.496+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:03.496+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:03.496+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:03.497+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.497+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:03.513+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.513+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.514+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:03.547+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.548+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:03.548+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:03.548+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:03.548+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:05.548+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.549+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.549+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:05.549+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:05.549+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.549+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.108+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:57.125+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.125+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:57.125+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.126+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:57.126+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:57.126+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:57.126+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.126+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:57.159+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.159+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:57.159+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:57.159+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:57.159+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:59.160+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.160+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:59.160+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:59.160+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:59.161+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.161+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:59.225+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.225+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.226+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:59.355+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.356+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:29:59.356+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:29:59.356+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:29:59.356+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:01.357+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:01.614+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:01.614+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:01.615+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:02.128+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:02.129+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:02.129+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:02.129+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:02.129+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:19 EDT 
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:39:21 EDT 
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
2013-10-14 17:39:23 EDT 
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:04.130+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:05.131+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:05.132+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:05.132+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:06.133+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:06.134+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:06.134+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:06.134+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:06.134+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:08.134+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.134+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:08.134+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:08.135+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:08.135+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.135+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:08.137+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:08.138+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.138+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:08.140+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.141+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:08.141+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:08.141+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:08.141+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:10.141+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:10.142+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:30:10.142+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:30:10.142+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:30:10.142+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:30:10.142+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:05.614+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:05.615+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.615+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.744+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.744+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:05.744+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:05.744+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:05.744+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:07.745+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:07.745+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:07.745+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:07.745+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:07.746+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:07.746+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:08.002+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:08.003+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:08.517+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:08.517+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:08.517+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:08.517+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:08.517+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:10.518+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:11.519+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:11.519+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:11.520+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:12.521+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:12.522+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:12.522+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:12.522+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:12.522+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:14.522+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.522+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:14.522+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:14.522+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:14.523+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.523+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:14.524+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.524+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.525+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:14.528+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.528+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:14.528+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:14.528+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:14.528+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:15.835+0000 [conn15] end connection 10.142.147.116:58681 (3 connections now open)
 m31002| 2013-10-14T21:30:12.966+0000 [conn7] end connection 10.142.147.116:59424 (1 connection now open)
 m31002| 2013-10-14T21:30:12.966+0000 [initandlisten] connection accepted from 10.142.147.116:59608 #8 (3 connections now open)
 m31002| 2013-10-14T21:31:13.582+0000 [initandlisten] connection accepted from 10.142.147.116:59663 #9 (3 connections now open)
 m31000| 2013-10-14T21:30:15.835+0000 [initandlisten] connection accepted from 10.142.147.116:58856 #16 (4 connections now open)
 m31000| 2013-10-14T21:30:16.529+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.529+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:16.529+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| 2013-10-14T21:38:50.501+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| 2013-10-14T21:38:50.501+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:16.529+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:16.530+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.530+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:16.534+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.534+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.535+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:16.544+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:16.544+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:13.582+0000 [initandlisten] connection accepted from 10.142.147.116:59701 #10 (4 connections now open)
 m31002| 2013-10-14T21:31:13.583+0000 [initandlisten] connection accepted from 10.142.147.116:59743 #11 (5 connections now open)
 m31002| 2013-10-14T21:31:13.583+0000 [conn9] end connection 10.142.147.116:59663 (4 connections now open)
 m31000| 2013-10-14T21:30:16.544+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:16.544+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:16.544+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.545+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:18.562+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:18.563+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.563+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:18.595+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.596+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:18.596+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:18.596+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:18.596+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.597+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.598+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.598+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.599+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.599+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.600+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.603+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.603+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:20.603+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:20.603+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:20.603+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.604+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:22.609+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:22.610+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.610+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:22.618+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.619+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:22.619+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:22.619+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:22.619+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:22.967+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:30:22.967+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31002 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31000" }
 m31000| 2013-10-14T21:30:22.967+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31002 is down (or slow to respond): 
2013-10-14 17:39:25 EDT 
 m31000| 2013-10-14T21:30:22.967+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31002 is now in state DOWN
2013-10-14 17:39:27 EDT 
 m31000| 2013-10-14T21:30:22.967+0000 [rsMgr] can't see a majority of the set, relinquishing primary
2013-10-14 17:39:30 EDT 
 m31000| 2013-10-14T21:30:22.967+0000 [rsMgr] replSet relinquishing primary state
2013-10-14 17:39:31 EDT 
 m31000| 2013-10-14T21:30:22.967+0000 [rsMgr] replSet SECONDARY
 m31000| 2013-10-14T21:30:22.967+0000 [rsMgr] replSet closing client sockets after relinquishing primary
 m31000| 2013-10-14T21:30:22.967+0000 [conn13] end connection 10.142.147.116:58358 (3 connections now open)
 m31000| 2013-10-14T21:30:22.967+0000 [conn1] end connection 127.0.0.1:59075 (2 connections now open)
 m31000| 2013-10-14T21:30:23.981+0000 [conn12] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.142.147.116:58357] 
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.620+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:24.637+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:24.638+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.638+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:24.670+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.671+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:24.671+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:24.671+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:24.671+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.672+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:26.737+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.737+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:26.738+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.738+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:26.738+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:26.738+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:26.739+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.739+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:26.867+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.868+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:26.868+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:26.868+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:26.868+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:28.869+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:28.869+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:28.869+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:28.869+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:28.870+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:28.870+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:29.127+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:29.128+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:29.128+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:29.641+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:29.641+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:29.641+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:29.641+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:29.641+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:29.641+0000 [rsMgr] replSet can't see a majority, will not try to elect self
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:31.642+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:32.643+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:32.643+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:32.644+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:33.645+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:33.645+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:33.645+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:33.646+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:33.646+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:34.969+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:30:34.969+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31002 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31000" }
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.648+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.650+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:35.655+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.656+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:35.656+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:35.656+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:35.656+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.657+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:37.662+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:37.663+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.663+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:37.671+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.672+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:37.672+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:37.672+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:37.672+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.673+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:39.690+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:39.691+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.691+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:39.723+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.724+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:39.724+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:39.724+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:39.724+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.725+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:41.790+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.790+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.791+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:41.920+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.921+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:41.921+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:41.921+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:41.921+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:43.921+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:43.921+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:43.921+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:43.922+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:43.922+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:43.922+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:44.179+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:44.180+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:44.180+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:44.692+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:44.693+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:44.693+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:44.693+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:44.693+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:46.694+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:46.971+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:30:47.695+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:47.695+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:47.696+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:48.697+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:48.697+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:48.697+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:48.698+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:48.698+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:50.698+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:34 EDT 
 m31000| 2013-10-14T21:30:50.698+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:36 EDT 
 m31000| 2013-10-14T21:30:50.698+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:39:37 EDT 
 m31000| 2013-10-14T21:30:50.698+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:50.699+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.699+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:50.700+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.700+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.701+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:50.704+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.704+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:50.704+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:50.704+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:50.704+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:52.705+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.705+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:52.705+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:52.705+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:52.706+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.706+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:52.710+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.710+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.711+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:52.720+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.720+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:52.720+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:52.720+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:52.720+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:54.721+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.721+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:54.721+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:54.721+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:54.722+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.722+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:54.738+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.738+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.739+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:54.772+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.772+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:54.773+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:54.773+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:54.773+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:56.773+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:56.773+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:56.773+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31000| 2013-10-14T21:30:56.773+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31000| 2013-10-14T21:30:56.774+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:56.774+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:56.774+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31000| 2013-10-14T21:30:56.774+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31000| 2013-10-14T21:30:58.973+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-14T21:31:13.584+0000 [initandlisten] connection accepted from 10.142.147.116:59027 #17 (2 connections now open)
 m31000| 2013-10-14T21:38:48.498+0000 [initandlisten] connection accepted from 10.142.147.116:59116 #18 (3 connections now open)
 m31000| 2013-10-14T21:31:35.588+0000 [conn16] end connection 10.142.147.116:58856 (0 connections now open)
 m31000| 2013-10-14T21:38:48.498+0000 [initandlisten] connection accepted from 10.142.147.116:59153 #19 (4 connections now open)
 m31000| 2013-10-14T21:38:48.498+0000 [initandlisten] connection accepted from 10.142.147.116:59193 #20 (4 connections now open)
 m31000| 2013-10-14T21:34:07.121+0000 [clientcursormon] mem (MB) res:73 virt:744
 m31000| 2013-10-14T21:38:48.498+0000 [clientcursormon]  mapped:128
 m31000| 2013-10-14T21:38:48.499+0000 [initandlisten] connection accepted from 10.142.147.116:59202 #21 (5 connections now open)
 m31000| 2013-10-14T21:38:48.499+0000 [clientcursormon]  connections:4
 m31000| 2013-10-14T21:38:48.499+0000 [conn17] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.142.147.116:59027] 
 m31000| 2013-10-14T21:38:48.499+0000 [clientcursormon]  replication threads:32
 m31000| 2013-10-14T21:38:48.499+0000 [initandlisten] connection accepted from 10.142.147.116:59204 #22 (6 connections now open)
 m31000| 2013-10-14T21:38:41.532+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31000| 2013-10-14T21:38:48.499+0000 [initandlisten] connection accepted from 10.142.147.116:59241 #23 (6 connections now open)
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] now exiting
 m31000| dbexit: 2013-10-14T21:38:48.499+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31000| 2013-10-14T21:38:48.499+0000 [initandlisten] connection accepted from 10.142.147.116:59284 #24 (7 connections now open)
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] closing listening socket: 8
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] closing listening socket: 9
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] closing listening socket: 12
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31000.sock
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31000| 2013-10-14T21:38:48.499+0000 [signalProcessingThread] shutdown: closing all files...
 m31000| 2013-10-14T21:38:48.502+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 thinks that we are down
 m31000| 2013-10-14T21:38:48.502+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is up
 m31000| 2013-10-14T21:38:48.502+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31001 is now in state SECONDARY
 m31000| 2013-10-14T21:38:48.503+0000 [signalProcessingThread] closeAllFiles() finished
 m31000| 2013-10-14T21:38:48.503+0000 [signalProcessingThread] shutdown: removing fs lock...
 m31002| 2013-10-14T21:30:22.967+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: ip-10-142-147-116:31000
 m31002| 2013-10-14T21:31:13.583+0000 [conn8] end connection 10.142.147.116:59608 (4 connections now open)
 m31002| 2013-10-14T21:31:13.583+0000 [rsBackgroundSync] replSet syncing to: ip-10-142-147-116:31000
 m31002| 2013-10-14T21:31:13.583+0000 [conn10] end connection 10.142.147.116:59701 (2 connections now open)
 m31002| 2013-10-14T21:30:23.837+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:13.583+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state SECONDARY
 m31002| 2013-10-14T21:31:13.584+0000 [rsMgr] not electing self, ip-10-142-147-116:31000 would veto with 'I don't think ip-10-142-147-116:31002 is electable'
 m31002| 2013-10-14T21:31:13.584+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.584+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.585+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:15.588+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:15.589+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.589+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:15.593+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.594+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:15.594+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:15.594+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:15.594+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:17.584+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:17.594+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.594+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.594+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:17.594+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:17.595+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.595+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.595+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.595+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:17.597+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:17.598+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.598+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.600+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.601+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:17.601+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:17.601+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:17.601+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:19.584+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:19.585+0000 [rsMgr] not electing self, ip-10-142-147-116:31000 would veto with 'I don't think ip-10-142-147-116:31002 is electable'
 m31002| 2013-10-14T21:31:19.601+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.601+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:19.601+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:19.601+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:19.602+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.602+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:19.606+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.606+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.607+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:19.616+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.616+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:19.616+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:19.616+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:19.616+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:21.585+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:21.617+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.617+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:21.617+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:21.617+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:21.618+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.618+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:21.634+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.634+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.635+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:21.668+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.668+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:21.668+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:21.668+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:21.668+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:23.585+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:23.669+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:23.669+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:23.669+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:23.669+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:23.670+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:23.670+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:23.734+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:23.735+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:23.864+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:39 EDT 
 m31002| 2013-10-14T21:31:23.865+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:41 EDT 
 m31002| 2013-10-14T21:31:23.865+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:49 EDT 
 m31002| 2013-10-14T21:31:23.865+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
2013-10-14 17:39:51 EDT 
 m31002| 2013-10-14T21:31:23.865+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:25.586+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:25.586+0000 [rsMgr] not electing self, ip-10-142-147-116:31000 would veto with 'I don't think ip-10-142-147-116:31002 is electable'
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:25.866+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:26.123+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:26.124+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:26.124+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:26.637+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:26.637+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:26.637+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:26.637+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:26.637+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:27.586+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:28.638+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:28.638+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:28.638+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:28.638+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:28.638+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:28.639+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:29.587+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:29.639+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:29.639+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:29.640+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:30.641+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:30.642+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:30.642+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:30.642+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:30.642+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:30.642+0000 [rsMgr] not electing self, ip-10-142-147-116:31000 would veto with 'I don't think ip-10-142-147-116:31002 is electable'
 m31002| 2013-10-14T21:31:31.587+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:32.642+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.642+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:32.642+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:32.642+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:32.643+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.643+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:32.644+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.644+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.645+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:32.648+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.648+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:32.648+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:32.648+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:32.648+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:33.588+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 thinks that we are down
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.649+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:34.654+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:34.655+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.655+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:34.663+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.664+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:34.664+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:34.664+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:34.664+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.665+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:36.682+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:36.683+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.683+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:36.715+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.716+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:36.716+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:36.716+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:36.716+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:38.716+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.717+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:38.717+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:38.717+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:38.717+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.717+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:38.782+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:38.783+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.783+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:38.911+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.912+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:38.912+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:38.912+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:38.912+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:40.913+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:41.170+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:41.171+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:41.171+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:41.683+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:41.684+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:41.684+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:41.684+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:41.684+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:43.583+0000 [rsBackgroundSync] Socket recv() timeout  10.142.147.116:31000
 m31002| 2013-10-14T21:31:43.583+0000 [rsBackgroundSync] SocketException: remote: 10.142.147.116:31000 error: 9001 socket exception [RECV_TIMEOUT] server [10.142.147.116:31000] 
 m31002| 2013-10-14T21:31:43.583+0000 [rsBackgroundSync] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:31:43.583+0000 [rsBackgroundSync] replSet sync source problem: 10276 DBClientBase::findN: transport error: ip-10-142-147-116:31000 ns: local.oplog.rs query: {}
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:43.685+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:44.686+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:44.686+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:44.687+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:45.588+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:31:45.588+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31000 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-142-147-116:31002" }
 m31002| 2013-10-14T21:31:45.588+0000 [rsHealthPoll] replset info ip-10-142-147-116:31000 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:45.688+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:45.688+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:45.688+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:45.688+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:45.688+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:47.689+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.689+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:47.689+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:47.689+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:47.690+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.690+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:47.691+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.691+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.692+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:47.695+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.695+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:47.695+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:47.695+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:47.695+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.697+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:49.702+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.702+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.703+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:52 EDT 
 m31002| 2013-10-14T21:31:49.712+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.713+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:49.713+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:49.713+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:49.713+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.714+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:51.731+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:51.732+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.732+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:51.764+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.765+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:51.765+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:51.765+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:51.765+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.766+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.767+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.767+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.768+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.768+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.769+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.772+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.772+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:53.772+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:53.772+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:53.772+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:55.589+0000 [MultiCommandJob] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:31:55.589+0000 [MultiCommandJob] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-142-147-116:31000 ns: admin.$cmd query: { replSetFresh: 1, set: "jstests_initsync1", opTime: new Date(5934726143927123969), who: "ip-10-142-147-116:31002", cfgver: 2, id: 2 }
 m31002| 2013-10-14T21:31:55.589+0000 [MultiCommandJob] dev caught dbexception on multiCommand ip-10-142-147-116:31000
 m31002| 2013-10-14T21:31:55.589+0000 [rsMgr] replSet freshest returns {}
 m31002| 2013-10-14T21:31:55.589+0000 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.773+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:55.778+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:55.779+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.779+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:55.787+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.788+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:55.788+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:55.788+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:55.788+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:56.591+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:31:57.593+0000 [MultiCommandJob] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:31:57.593+0000 [MultiCommandJob] dev caught dbexception on multiCommand ip-10-142-147-116:31000
 m31002| 2013-10-14T21:31:57.593+0000 [rsHealthPoll] replSet info ip-10-142-147-116:31000 is down (or slow to respond): 
 m31002| 2013-10-14T21:31:57.593+0000 [rsHealthPoll] replSet member ip-10-142-147-116:31000 is now in state DOWN
 m31002| 2013-10-14T21:31:57.593+0000 [rsMgr] replSet freshest returns {}
 m31002| 2013-10-14T21:31:57.593+0000 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.789+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:57.806+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:57.807+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.807+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:57.839+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.840+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:57.840+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:57.840+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:57.840+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:59.840+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:59.840+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:59.840+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:59.840+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:59.841+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:59.841+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:59.906+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:59.906+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:31:59.907+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:00.036+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:00.037+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:00.037+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:00.037+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:00.037+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:02.037+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.038+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:02.038+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:02.038+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:02.038+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.038+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:02.295+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:02.296+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.296+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:02.809+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.809+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:02.809+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:02.809+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:02.809+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:04.810+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:04.810+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:04.810+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:04.810+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:04.811+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:04.811+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:05.811+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:05.812+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:06.813+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:06.814+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:06.814+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:06.814+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:06.814+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:07.598+0000 [MultiCommandJob] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:32:07.598+0000 [MultiCommandJob] dev caught dbexception on multiCommand ip-10-142-147-116:31000
 m31002| 2013-10-14T21:32:07.598+0000 [rsMgr] replSet freshest returns {}
 m31002| 2013-10-14T21:32:07.598+0000 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.815+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:08.817+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:08.818+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.818+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:08.820+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.820+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:08.821+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:08.821+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:08.821+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:10.821+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.821+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:10.821+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:10.821+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:10.822+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.822+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:10.826+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.826+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.827+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:10.836+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.836+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:10.836+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:10.836+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:10.836+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:12.837+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.837+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:12.837+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:12.837+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:12.838+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.838+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
2013-10-14 17:39:54 EDT 
 m31002| 2013-10-14T21:32:12.854+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
2013-10-14 17:39:56 EDT 
 m31002| 2013-10-14T21:32:12.854+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.855+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:12.888+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.889+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:12.889+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:12.889+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:12.889+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:14.889+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:14.889+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:14.889+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:14.889+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:14.890+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:14.890+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:14.954+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:14.955+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:15.084+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:15.085+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:15.085+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:15.085+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:15.085+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:17.086+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.086+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:17.086+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:17.086+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:17.086+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.087+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:17.343+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.343+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.344+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:17.600+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:32:17.857+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.858+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:17.858+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:17.858+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:17.858+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:19.858+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:19.858+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:19.858+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:19.858+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:19.859+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:19.859+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] replset info ip-10-142-147-116:31001 heartbeat failed, retrying
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-14T21:32:20.860+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-14T21:32:20.861+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:20.861+0000 [rsHealthPoll] couldn't connect to ip-10-142-147-116:31001: couldn't connect to server ip-10-142-147-116:31001
 m31002| 2013-10-14T21:32:21.862+0000 [rsHealthPoll] warning: Failed to connect to 10.142.147.116, reason: errno:111 Connection refused
 m31002| 2013-10-14T21:32:27.601+0000 [MultiCommandJob] DBClientCursor::init call() failed
 m31002| 2013-10-14T21:32:30.682+0000 [initandlisten] connection accepted from 10.142.147.116:60067 #12 (3 connections now open)
 m31002| 2013-10-14T21:34:28.801+0000 [clientcursormon] mem (MB) res:58 virt:666
 m31002| 2013-10-14T21:38:55.475+0000 [MultiCommandJob] dev caught dbexception on multiCommand ip-10-142-147-116:31000
 m31002| 2013-10-14T21:38:55.475+0000 [initandlisten] connection accepted from 10.142.147.116:60077 #13 (4 connections now open)
 m31002| 2013-10-14T21:38:55.476+0000 [rsMgr] replSet freshest returns {}
 m31002| 2013-10-14T21:38:48.500+0000 [conn11] end connection 10.142.147.116:59743 (1 connection now open)
 m31002| 2013-10-14T21:38:55.476+0000 [rsMgr] replSet info electSelf 2
 m31002| 2013-10-14T21:38:55.476+0000 [conn12] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.142.147.116:60067] 
 m31002| 2013-10-14T21:38:53.533+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31002| 2013-10-14T21:38:55.475+0000 [clientcursormon]  mapped:64
 m31002| 2013-10-14T21:38:55.476+0000 [clientcursormon]  connections:2
 m31002| 2013-10-14T21:38:55.476+0000 [signalProcessingThread] now exiting
 m31002| 2013-10-14T21:38:55.476+0000 [clientcursormon]  replication threads:32
 m31002| dbexit: 2013-10-14T21:38:55.476+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31002| 2013-10-14T21:38:55.476+0000 [signalProcessingThread] closing listening socket: 14
 m31002| 2013-10-14T21:38:55.476+0000 [rsHealthPoll] warning: Failed to connect to 10.14