Home > mci_0.9_linux_64_debug_duroff #1340 > replicasets_0 > initial_sync1.js
Plain Text
2013-10-14 20:11:57 EDT 
MongoDB shell version: 2.5.3-pre-
false
1. Bring up set
ReplSetTest Starting Set
ReplSetTest n is : 0
ReplSetTest n: 0 ports: [ 31000, 31001 ]    31000 number
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31000,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : undefined,
    "pathOpts" : {
        "node" : 0,
        "set" : "jstests_initsync1"
    }
}
ReplSetTest Starting....
Resetting db path '/data/db/jstests_initsync1-0'
2013-10-15T00:11:57.488+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-0 --setParameter enableTestCommands=1
2013-10-15T00:11:57.490+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31000| 2013-10-15T00:11:57.534+0000 ** WARNING: --rest is specified without --httpinterface,
 m31000| 2013-10-15T00:11:57.534+0000 **          enabling http interface
 m31000| note: noprealloc may hurt performance in many applications
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] MongoDB starting : pid=5704 port=31000 dbpath=/data/db/jstests_initsync1-0 64-bit host=ip-10-159-32-121
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] _DEBUG build (which is slower)
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] 
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] **       Not recommended for production.
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] 
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] db version v2.5.3-pre-
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] git version: d1c2ae72c4829ed77cb6b7fe42e486910483caff
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] allocator: tcmalloc
 m31000| 2013-10-15T00:11:57.541+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-0", noprealloc: true, oplogSize: 40, port: 31000, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31000| 2013-10-15T00:11:57.575+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31000| 2013-10-15T00:11:57.575+0000 [initandlisten] opening db:  admin
 m31000| 2013-10-15T00:11:57.576+0000 [initandlisten] 
 m31000| debug have W lock but w would suffice for command create
 m31000| 2013-10-15T00:11:57.576+0000 [initandlisten] opening db:  local
 m31000| 2013-10-15T00:11:57.576+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.ns, filling with zeroes...
 m31000| 2013-10-15T00:11:57.576+0000 [FileAllocator] creating directory /data/db/jstests_initsync1-0/_tmp
 m31000| 2013-10-15T00:11:57.588+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.ns, size: 16MB,  took 0.007 secs
 m31000| 2013-10-15T00:11:57.590+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.0, filling with zeroes...
 m31000| 2013-10-15T00:11:57.598+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.0, size: 16MB,  took 0.007 secs
 m31000| 2013-10-15T00:11:57.598+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1-0/local.0 n:0
 m31000| 2013-10-15T00:11:57.598+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31000| 2013-10-15T00:11:57.598+0000 [initandlisten] waiting for connections on port 31000
 m31000| 2013-10-15T00:11:57.598+0000 [websvr] admin web console waiting for connections on port 32000
 m31000| 2013-10-15T00:11:57.601+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31000| 2013-10-15T00:11:57.601+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31000| 2013-10-15T00:11:57.692+0000 [initandlisten] connection accepted from 127.0.0.1:42610 #1 (1 connection now open)
[ connection to ip-10-159-32-121:31000 ]
ReplSetTest n is : 1
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31001,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : undefined,
    "pathOpts" : {
        "node" : 1,
        "set" : "jstests_initsync1"
    }
}
ReplSetTest Starting....
Resetting db path '/data/db/jstests_initsync1-1'
2013-10-15T00:11:57.702+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-1 --setParameter enableTestCommands=1
2013-10-15T00:11:57.704+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31001| 2013-10-15T00:11:57.748+0000 ** WARNING: --rest is specified without --httpinterface,
 m31001| 2013-10-15T00:11:57.748+0000 **          enabling http interface
 m31001| note: noprealloc may hurt performance in many applications
 m31001| 2013-10-15T00:11:57.754+0000 [initandlisten] MongoDB starting : pid=5753 port=31001 dbpath=/data/db/jstests_initsync1-1 64-bit host=ip-10-159-32-121
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] _DEBUG build (which is slower)
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] 
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] **       Not recommended for production.
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] 
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] db version v2.5.3-pre-
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] git version: d1c2ae72c4829ed77cb6b7fe42e486910483caff
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] allocator: tcmalloc
 m31001| 2013-10-15T00:11:57.755+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31001| 2013-10-15T00:11:57.776+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31001| 2013-10-15T00:11:57.776+0000 [initandlisten] opening db:  admin
 m31001| 2013-10-15T00:11:57.777+0000 [initandlisten] 
 m31001| debug have W lock but w would suffice for command create
 m31001| 2013-10-15T00:11:57.777+0000 [initandlisten] opening db:  local
 m31001| 2013-10-15T00:11:57.777+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.ns, filling with zeroes...
 m31001| 2013-10-15T00:11:57.777+0000 [FileAllocator] creating directory /data/db/jstests_initsync1-1/_tmp
 m31001| 2013-10-15T00:11:57.783+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.ns, size: 16MB,  took 0.004 secs
 m31001| 2013-10-15T00:11:57.785+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.0, filling with zeroes...
 m31001| 2013-10-15T00:11:57.788+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.0, size: 16MB,  took 0.002 secs
 m31001| 2013-10-15T00:11:57.788+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1-1/local.0 n:0
 m31001| 2013-10-15T00:11:57.788+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31001| 2013-10-15T00:11:57.789+0000 [initandlisten] waiting for connections on port 31001
 m31001| 2013-10-15T00:11:57.789+0000 [websvr] admin web console waiting for connections on port 32001
 m31001| 2013-10-15T00:11:57.791+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31001| 2013-10-15T00:11:57.792+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31001| 2013-10-15T00:11:57.905+0000 [initandlisten] connection accepted from 127.0.0.1:38864 #1 (1 connection now open)
[
    connection to ip-10-159-32-121:31000,
    connection to ip-10-159-32-121:31001
]
{
    "replSetInitiate" : {
        "_id" : "jstests_initsync1",
        "members" : [
            {
                "_id" : 0,
                "host" : "ip-10-159-32-121:31000"
            },
            {
                "_id" : 1,
                "host" : "ip-10-159-32-121:31001"
            }
        ]
    }
}
 m31000| 2013-10-15T00:11:57.909+0000 [conn1] replSet replSetInitiate admin command received from client
 m31000| 2013-10-15T00:11:57.911+0000 [conn1] replSet replSetInitiate config object parses ok, 2 members specified
 m31001| 2013-10-15T00:11:57.913+0000 [initandlisten] connection accepted from 10.159.32.121:48772 #2 (2 connections now open)
 m31000| 2013-10-15T00:11:57.914+0000 [conn1] replSet replSetInitiate all members seem up
 m31000| 2013-10-15T00:11:57.914+0000 [conn1] ******
 m31000| 2013-10-15T00:11:57.914+0000 [conn1] creating replication oplog of size: 40MB...
 m31000| 2013-10-15T00:11:57.915+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/local.1, filling with zeroes...
 m31000| 2013-10-15T00:11:57.924+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/local.1, size: 64MB,  took 0.009 secs
 m31000| 2013-10-15T00:11:57.924+0000 [conn1] datafileheader::init initializing /data/db/jstests_initsync1-0/local.1 n:1
 m31000| 2013-10-15T00:11:57.970+0000 [conn1] ******
 m31000| 2013-10-15T00:11:57.970+0000 [conn1] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 1, members: [ { _id: 0, host: "ip-10-159-32-121:31000" }, { _id: 1, host: "ip-10-159-32-121:31001" } ] }
 m31000| 2013-10-15T00:11:57.971+0000 [conn1] replSet saveConfigLocally done
 m31000| 2013-10-15T00:11:57.971+0000 [conn1] replSet replSetInitiate config now saved locally.  Should come online in about a minute.
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}
2013-10-14 20:12:01 EDT 
2013-10-15T00:12:01.769+0000 [conn14] end connection 127.0.0.1:57871 (0 connections now open)
2013-10-15T00:12:02.369+0000 [initandlisten] connection accepted from 127.0.0.1:57883 #15 (1 connection now open)
2013-10-15T00:12:02.371+0000 [conn15] end connection 127.0.0.1:57883 (0 connections now open)
 m31000| 2013-10-15T00:11:58.601+0000 [rsStart] replSet I am ip-10-159-32-121:31000
 m31000| 2013-10-15T00:11:58.601+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31001
 m31000| 2013-10-15T00:11:58.601+0000 [rsStart] replSet STARTUP2
 m31000| 2013-10-15T00:11:58.602+0000 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
 m31000| 2013-10-15T00:11:58.602+0000 [rsSync] replSet SECONDARY
 m31000| 2013-10-15T00:11:58.604+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
 m31000| 2013-10-15T00:11:58.605+0000 [rsSyncNotifier] have free list for local.$freelist
 m31000| 2013-10-15T00:11:58.606+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
 m31001| 2013-10-15T00:11:58.792+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31001| 2013-10-15T00:11:59.792+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31000| 2013-10-15T00:12:00.602+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is up
 m31000| 2013-10-15T00:12:00.602+0000 [rsMgr] replSet freshest returns { startupStatus: 3, info: "run rs.initiate(...) if not yet done for the set", ok: 0.0, errmsg: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)" }
 m31000| 2013-10-15T00:12:00.602+0000 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
 m31001| 2013-10-15T00:12:00.792+0000 [rsStart] trying to contact ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:00.794+0000 [initandlisten] connection accepted from 10.159.32.121:41698 #2 (2 connections now open)
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] replSet load config ok from ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] replSet I am ip-10-159-32-121:31001
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] replSet got config version 1 from a remote, saving locally
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 1, members: [ { _id: 0, host: "ip-10-159-32-121:31000" }, { _id: 1, host: "ip-10-159-32-121:31001" } ] }
 m31001| 2013-10-15T00:12:00.795+0000 [rsStart] replSet saveConfigLocally done
 m31001| 2013-10-15T00:12:00.796+0000 [rsStart] replSet STARTUP2
 m31001| 2013-10-15T00:12:00.796+0000 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
 m31001| 2013-10-15T00:12:00.796+0000 [rsSync] ******
 m31001| 2013-10-15T00:12:00.796+0000 [rsSync] creating replication oplog of size: 40MB...
 m31001| 2013-10-15T00:12:00.796+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/local.1, filling with zeroes...
 m31001| 2013-10-15T00:12:00.821+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/local.1, size: 64MB,  took 0.024 secs
 m31001| 2013-10-15T00:12:00.821+0000 [rsSync] datafileheader::init initializing /data/db/jstests_initsync1-1/local.1 n:1
 m31001| 2013-10-15T00:12:00.850+0000 [rsSync] ******
 m31001| 2013-10-15T00:12:00.851+0000 [rsSync] replSet initial sync pending
 m31001| 2013-10-15T00:12:00.851+0000 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
 m31001| 2013-10-15T00:12:00.853+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
 m31001| 2013-10-15T00:12:00.853+0000 [rsSyncNotifier] have free list for local.$freelist
 m31001| 2013-10-15T00:12:00.855+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
 m31000| 2013-10-15T00:12:02.602+0000 [rsHealthPoll] replset info ip-10-159-32-121:31001 thinks that we are down
 m31000| 2013-10-15T00:12:02.602+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state STARTUP2
2013-10-14 20:12:17 EDT 
 m31000| 2013-10-15T00:12:02.603+0000 [rsMgr] not electing self, ip-10-159-32-121:31001 would veto with 'I don't think ip-10-159-32-121:31000 is electable'
 m31001| 2013-10-15T00:12:02.795+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is up
 m31001| 2013-10-15T00:12:02.796+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is now in state SECONDARY
 m31000| 2013-10-15T00:12:08.604+0000 [rsMgr] replSet info electSelf 0
 m31001| 2013-10-15T00:12:08.604+0000 [conn2] replSet received elect msg { replSetElect: 1, set: "jstests_initsync1", who: "ip-10-159-32-121:31000", whoid: 0, cfgver: 1, round: ObjectId('525c8858def5fcb594a1cabc') }
 m31001| 2013-10-15T00:12:08.604+0000 [conn2] replSet RECOVERING
 m31001| 2013-10-15T00:12:08.604+0000 [conn2] replSet info voting yea for ip-10-159-32-121:31000 (0)
 m31000| 2013-10-15T00:12:08.604+0000 [rsMgr] replSet elect res: { vote: 1, round: ObjectId('525c8858def5fcb594a1cabc'), ok: 1.0 }
 m31000| 2013-10-15T00:12:09.603+0000 [rsMgr] replSet PRIMARY
 m31000| 2013-10-15T00:12:10.604+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state RECOVERING
 m31001| 2013-10-15T00:12:10.797+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is now in state PRIMARY
 m31001| 2013-10-15T00:12:16.851+0000 [rsSync] replSet initial sync pending
 m31001| 2013-10-15T00:12:16.851+0000 [rsSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:16.852+0000 [initandlisten] connection accepted from 10.159.32.121:41701 #3 (3 connections now open)
 m31001| 2013-10-15T00:12:16.853+0000 [rsSync] build index on: local.replset.minvalid properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.replset.minvalid", "name" : "_id_" }
 m31001| 2013-10-15T00:12:16.853+0000 [rsSync] build index done.  scanned 0 total records. 0 secs
 m31001| 2013-10-15T00:12:16.853+0000 [rsSync] replSet initial sync drop all databases
 m31001| 2013-10-15T00:12:16.853+0000 [rsSync] dropAllDatabasesExceptLocal 1
 m31001| 2013-10-15T00:12:16.853+0000 [rsSync] replSet initial sync clone all databases
 m31001| 2013-10-15T00:12:16.854+0000 [rsSync] replSet initial sync cloning db: admin
 m31000| 2013-10-15T00:12:16.854+0000 [initandlisten] connection accepted from 10.159.32.121:41702 #4 (4 connections now open)
 m31001| 2013-10-15T00:12:16.855+0000 [rsSync] replSet initial sync data copy, starting syncup
 m31001| 2013-10-15T00:12:16.855+0000 [rsSync] oplog sync 1 of 3
 m31001| 2013-10-15T00:12:16.855+0000 [rsSync] oplog sync 2 of 3
 m31001| 2013-10-15T00:12:16.855+0000 [rsSync] replSet initial sync building indexes
 m31001| 2013-10-15T00:12:16.855+0000 [rsSync] replSet initial sync cloning indexes for : admin
 m31001| 2013-10-15T00:12:16.857+0000 [rsSync] oplog sync 3 of 3
 m31001| 2013-10-15T00:12:16.857+0000 [rsSync] replSet initial sync finishing up
 m31000| 2013-10-15T00:12:16.857+0000 [conn4] end connection 10.159.32.121:41702 (3 connections now open)
 m31001| 2013-10-15T00:12:16.904+0000 [rsSync] replSet set minValid=525c884d:b
 m31001| 2013-10-15T00:12:16.910+0000 [rsSync] replSet initial sync done
 m31000| 2013-10-15T00:12:16.911+0000 [conn3] end connection 10.159.32.121:41701 (2 connections now open)
 m31001| 2013-10-15T00:12:17.797+0000 [rsBackgroundSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:17.798+0000 [initandlisten] connection accepted from 10.159.32.121:41703 #5 (3 connections now open)
 m31001| 2013-10-15T00:12:17.798+0000 [rsBackgroundSync] replSet remoteOldestOp:    Oct 15 00:11:57 525c884d:b
 m31001| 2013-10-15T00:12:17.798+0000 [rsBackgroundSync] replSet lastOpTimeFetched: Oct 15 00:11:57 525c884d:b
 m31000| 2013-10-15T00:12:17.799+0000 [initandlisten] connection accepted from 10.159.32.121:41704 #6 (4 connections now open)
 m31001| 2013-10-15T00:12:17.911+0000 [rsSync] replSet SECONDARY
2. Insert some data
2013-10-14 20:12:20 EDT 
 m31000| 2013-10-15T00:12:17.996+0000 [conn1] opening db:  foo
 m31000| 2013-10-15T00:12:17.996+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/foo.ns, filling with zeroes...
 m31000| 2013-10-15T00:12:18.003+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/foo.ns, size: 16MB,  took 0.006 secs
 m31000| 2013-10-15T00:12:18.005+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-0/foo.0, filling with zeroes...
 m31000| 2013-10-15T00:12:18.012+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-0/foo.0, size: 16MB,  took 0.007 secs
 m31000| 2013-10-15T00:12:18.012+0000 [conn1] datafileheader::init initializing /data/db/jstests_initsync1-0/foo.0 n:0
 m31000| 2013-10-15T00:12:18.014+0000 [conn1] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31000| 2013-10-15T00:12:18.014+0000 [conn1] have free list for foo.$freelist
 m31000| 2013-10-15T00:12:18.015+0000 [conn1] build index done.  scanned 0 total records. 0.001 secs
 m31001| 2013-10-15T00:12:18.022+0000 [repl prefetch worker] _DEBUG ReadContext db wasn't open, will try to open foo.bar
 m31001| 2013-10-15T00:12:18.022+0000 [repl prefetch worker] _DEBUG ReadContext db wasn't open, will try to open foo.bar
 m31001| 2013-10-15T00:12:18.022+0000 [repl prefetch worker] opening db:  foo
 m31001| 2013-10-15T00:12:18.024+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/foo.ns, filling with zeroes...
 m31001| 2013-10-15T00:12:18.039+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/foo.ns, size: 16MB,  took 0.014 secs
 m31001| 2013-10-15T00:12:18.041+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1-1/foo.0, filling with zeroes...
 m31001| 2013-10-15T00:12:18.052+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1-1/foo.0, size: 16MB,  took 0.01 secs
 m31001| 2013-10-15T00:12:18.052+0000 [repl writer worker 1] datafileheader::init initializing /data/db/jstests_initsync1-1/foo.0 n:0
 m31001| 2013-10-15T00:12:18.053+0000 [repl writer worker 1] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31001| 2013-10-15T00:12:18.054+0000 [repl writer worker 1] have free list for foo.$freelist
 m31001| 2013-10-15T00:12:18.055+0000 [repl writer worker 1] build index done.  scanned 0 total records. 0.002 secs
 m31000| 2013-10-15T00:12:18.605+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state SECONDARY
 m31000| 2013-10-15T00:12:18.857+0000 [slaveTracking] build index on: local.slaves properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.slaves", "name" : "_id_" }
 m31000| 2013-10-15T00:12:18.857+0000 [slaveTracking] build index done.  scanned 0 total records. 0 secs
total in foo: 10000
4. Make sure synced
ReplSetTest awaitReplication: starting: timestamp for primary, ip-10-159-32-121:31000, is Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795939, 2259) but latest is Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: last oplog entry (of 6353) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795939, 2259),  "h" : NumberLong("5597290987276098543"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c88621b8ae164b6e4b7c0"),  "date" : ISODate("2013-10-15T00:12:18.990Z"),  "x" : 6351,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
 m31001| 2013-10-15T00:12:20.871+0000 [conn1] query local.oplog.rs query: { query: {}, orderby: { $natural: -1.0 } } ntoreturn:1 ntoskip:0 keyUpdates:0 locks(micros) r:123 nreturned:1 reslen:167 120ms
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795940, 501) but latest is Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: last oplog entry (of 9299) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795940, 501),  "h" : NumberLong("6120751267583951560"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c88631b8ae164b6e4c342"),  "date" : ISODate("2013-10-15T00:12:19.428Z"),  "x" : 9297,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795940, 1203)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is synced
ReplSetTest awaitReplication: finished: all 1 secondaries synced at timestamp Timestamp(1381795940, 1203)
5. Freeze #2
 m31001| 2013-10-15T00:12:21.081+0000 [conn1] replSet info 'freezing' for 999999 seconds
6. Bring up #3
2013-10-15T00:12:21.089+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --port 31002 --dbpath /data/db/jstests_initsync1 --noprealloc --smallfiles --oplogSize 2 --nohttpinterface --replSet jstests_initsync1 --setParameter enableTestCommands=1
2013-10-15T00:12:21.091+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31002| note: noprealloc may hurt performance in many applications
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] MongoDB starting : pid=5833 port=31002 dbpath=/data/db/jstests_initsync1 64-bit host=ip-10-159-32-121
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] _DEBUG build (which is slower)
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] 
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] **       Not recommended for production.
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] 
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] db version v2.5.3-pre-
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] git version: d1c2ae72c4829ed77cb6b7fe42e486910483caff
 m31002| 2013-10-15T00:12:21.145+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31002| 2013-10-15T00:12:21.146+0000 [initandlisten] allocator: tcmalloc
 m31002| 2013-10-15T00:12:21.146+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1", nohttpinterface: true, noprealloc: true, oplogSize: 2, port: 31002, replSet: "jstests_initsync1", setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31002| 2013-10-15T00:12:21.152+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31002| 2013-10-15T00:12:21.152+0000 [initandlisten] opening db:  admin
 m31002| 2013-10-15T00:12:21.153+0000 [initandlisten] 
 m31002| debug have W lock but w would suffice for command create
 m31002| 2013-10-15T00:12:21.153+0000 [initandlisten] opening db:  local
 m31002| 2013-10-15T00:12:21.154+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/local.ns, filling with zeroes...
 m31002| 2013-10-15T00:12:21.154+0000 [FileAllocator] creating directory /data/db/jstests_initsync1/_tmp
 m31002| 2013-10-15T00:12:21.193+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/local.ns, size: 16MB,  took 0.005 secs
 m31002| 2013-10-15T00:12:21.195+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/local.0, filling with zeroes...
 m31002| 2013-10-15T00:12:21.201+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/local.0, size: 16MB,  took 0.005 secs
 m31002| 2013-10-15T00:12:21.201+0000 [initandlisten] datafileheader::init initializing /data/db/jstests_initsync1/local.0 n:0
 m31002| 2013-10-15T00:12:21.201+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31002| 2013-10-15T00:12:21.202+0000 [initandlisten] waiting for connections on port 31002
 m31002| 2013-10-15T00:12:21.204+0000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
 m31002| 2013-10-15T00:12:21.204+0000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
 m31002| 2013-10-15T00:12:21.292+0000 [initandlisten] connection accepted from 127.0.0.1:59238 #1 (1 connection now open)
 m31000| 2013-10-15T00:12:21.294+0000 [conn1] replSet replSetReconfig config object parses ok, 3 members specified
 m31002| 2013-10-15T00:12:21.296+0000 [initandlisten] connection accepted from 10.159.32.121:36422 #2 (2 connections now open)
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] replSet replSetReconfig [2]
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-159-32-121:31000" }, { _id: 1, host: "ip-10-159-32-121:31001" }, { _id: 2, host: "ip-10-159-32-121:31002" } ] }
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] replSet saveConfigLocally done
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] replSet info : additive change to configuration
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] starting rsHealthPoll for ip-10-159-32-121:31002
 m31000| 2013-10-15T00:12:21.296+0000 [conn1] replSet replSetReconfig new config saved locally
 m31000| 2013-10-15T00:12:21.298+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is up
 m31002| 2013-10-15T00:12:21.301+0000 [conn1] _DEBUG ReadContext db wasn't open, will try to open foo.bar
 m31002| 2013-10-15T00:12:21.301+0000 [conn1] opening db:  foo
{
    "_id" : "jstests_initsync1",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-159-32-121:31001"
        }
    ]
}
null
waiting 3
2013-10-14 20:12:22 EDT 
 m31002| 2013-10-15T00:12:22.205+0000 [rsStart] trying to contact ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:22.206+0000 [initandlisten] connection accepted from 10.159.32.121:41712 #7 (5 connections now open)
 m31002| 2013-10-15T00:12:22.207+0000 [rsStart] replSet load config ok from ip-10-159-32-121:31000
 m31002| 2013-10-15T00:12:22.208+0000 [rsStart] replSet I am ip-10-159-32-121:31002
 m31002| 2013-10-15T00:12:22.208+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31001
 m31002| 2013-10-15T00:12:22.208+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31000
 m31002| 2013-10-15T00:12:22.208+0000 [rsStart] replSet got config version 2 from a remote, saving locally
 m31002| 2013-10-15T00:12:22.208+0000 [rsStart] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-159-32-121:31000" }, { _id: 1, host: "ip-10-159-32-121:31001" }, { _id: 2, host: "ip-10-159-32-121:31002" } ] }
 m31002| 2013-10-15T00:12:22.209+0000 [rsStart] replSet saveConfigLocally done
 m31002| 2013-10-15T00:12:22.210+0000 [rsStart] replSet STARTUP2
 m31002| 2013-10-15T00:12:22.210+0000 [rsSync] ******
 m31002| 2013-10-15T00:12:22.210+0000 [rsSync] creating replication oplog of size: 2MB...
 m31002| 2013-10-15T00:12:22.269+0000 [rsSync] ******
 m31002| 2013-10-15T00:12:22.269+0000 [rsSync] replSet initial sync pending
 m31002| 2013-10-15T00:12:22.269+0000 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync
 m31002| 2013-10-15T00:12:22.271+0000 [rsSyncNotifier] build index on: local.me properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.me", "name" : "_id_" }
 m31002| 2013-10-15T00:12:22.272+0000 [rsSyncNotifier] have free list for local.$freelist
 m31002| 2013-10-15T00:12:22.273+0000 [rsSyncNotifier] build index done.  scanned 0 total records. 0.001 secs
{
    "_id" : "jstests_initsync1",
    "version" : 1,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-159-32-121:31001"
        }
    ]
}
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-159-32-121:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-159-32-121:31002"
        }
    ]
}
 m31001| 2013-10-15T00:12:22.799+0000 [rsMgr] replset msgReceivedNewConfig version: version: 2
 m31001| 2013-10-15T00:12:22.800+0000 [rsMgr] replSet info saving a newer config version to local.system.replset: { _id: "jstests_initsync1", version: 2, members: [ { _id: 0, host: "ip-10-159-32-121:31000" }, { _id: 1, host: "ip-10-159-32-121:31001" }, { _id: 2, host: "ip-10-159-32-121:31002" } ] }
 m31001| 2013-10-15T00:12:22.800+0000 [rsMgr] replSet saveConfigLocally done
 m31001| 2013-10-15T00:12:22.800+0000 [rsMgr] replSet info : additive change to configuration
 m31001| 2013-10-15T00:12:22.800+0000 [rsMgr] starting rsHealthPoll for ip-10-159-32-121:31002
 m31001| 2013-10-15T00:12:22.800+0000 [rsMgr] replSet replSetReconfig new config saved locally
 m31002| 2013-10-15T00:12:22.801+0000 [initandlisten] connection accepted from 10.159.32.121:36426 #3 (3 connections now open)
 m31001| 2013-10-15T00:12:22.801+0000 [rsHealthPoll] replset info ip-10-159-32-121:31002 thinks that we are down
 m31001| 2013-10-15T00:12:22.801+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is up
 m31001| 2013-10-15T00:12:22.801+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state STARTUP2
 m31000| 2013-10-15T00:12:23.299+0000 [rsHealthPoll] replset info ip-10-159-32-121:31002 thinks that we are down
 m31000| 2013-10-15T00:12:23.299+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state STARTUP2
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-159-32-121:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-159-32-121:31002"
        }
    ]
}
{
    "_id" : "jstests_initsync1",
    "version" : 2,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 1,
            "host" : "ip-10-159-32-121:31001"
        },
        {
            "_id" : 2,
            "host" : "ip-10-159-32-121:31002"
        }
    ]
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:23Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : -1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:23Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : -1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:22Z"),
            "pingMs" : 0
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 2,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
2013-10-14 20:12:24 EDT 
 m31002| 2013-10-15T00:12:24.209+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is up
 m31002| 2013-10-15T00:12:24.209+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is now in state PRIMARY
 m31001| 2013-10-15T00:12:24.209+0000 [initandlisten] connection accepted from 10.159.32.121:48791 #3 (3 connections now open)
 m31002| 2013-10-15T00:12:24.209+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is up
 m31002| 2013-10-15T00:12:24.209+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state SECONDARY
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:24Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 0,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:24Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:23Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 0,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:24Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:22Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 3,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:25Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 1,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:24Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:25Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 1,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:24Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:24Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 4,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
2013-10-14 20:12:26 EDT 
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:26Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 2,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:26Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:25Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 2,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:26Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:24Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 5,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
function () {
    var status = admin_s2.runCommand({replSetGetStatus:1});
    printjson(status);
    return status.members &&
      (status.members[2].state == 3 || status.members[2].state == 2);
  }
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:27Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 3,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:26Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:27Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 3,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:26Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:26Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 6,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
2013-10-15T00:12:27.834+0000 [initandlisten] connection accepted from 127.0.0.1:57959 #16 (1 connection now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:28Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 4,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:28Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:27Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 4,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:28Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:26Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 7,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31001| 2013-10-15T00:12:28.607+0000 [conn2] end connection 10.159.32.121:48772 (2 connections now open)
 m31001| 2013-10-15T00:12:28.608+0000 [initandlisten] connection accepted from 10.159.32.121:48800 #4 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:29Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 5,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:28Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:29Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 5,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:28Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:28Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 8,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
2013-10-14 20:12:30 EDT 
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:30Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 6,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:30Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:29Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 6,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:30Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:28Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 9,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31000| 2013-10-15T00:12:30.801+0000 [conn2] end connection 10.159.32.121:41698 (4 connections now open)
 m31000| 2013-10-15T00:12:30.802+0000 [initandlisten] connection accepted from 10.159.32.121:41728 #8 (5 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:31Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 7,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:30Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:31Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 7,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:30Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:30Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 10,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
2013-10-14 20:12:32 EDT 
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:32Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 8,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:32Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:31Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 8,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:32Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:30Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 11,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:33Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 9,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:32Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:33Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 9,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:32Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:32Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 12,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
2013-10-14 20:12:34 EDT 
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:34Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 10,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:34Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:33Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 10,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:34Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:32Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 13,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-15T00:12:35.301+0000 [conn2] end connection 10.159.32.121:36422 (2 connections now open)
 m31002| 2013-10-15T00:12:35.302+0000 [initandlisten] connection accepted from 10.159.32.121:36444 #4 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:35Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 11,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:34Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:35Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 11,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:34Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:34Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 14,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
waiting 4
2013-10-14 20:12:36 EDT 
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:36Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 12,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:36Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:35Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 12,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:36Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:34Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 15,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-15T00:12:36.804+0000 [conn3] end connection 10.159.32.121:36426 (2 connections now open)
 m31002| 2013-10-15T00:12:36.804+0000 [initandlisten] connection accepted from 10.159.32.121:36447 #5 (3 connections now open)
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:37Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 13,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:36Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:37Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 13,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:36Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:36Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 16,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync need a member to be primary or secondary to do our initial sync",
            "self" : true
        }
    ],
    "ok" : 1
}
2013-10-14 20:12:38 EDT 
 m31000| 2013-10-15T00:12:38.212+0000 [conn7] end connection 10.159.32.121:41712 (4 connections now open)
 m31000| 2013-10-15T00:12:38.212+0000 [initandlisten] connection accepted from 10.159.32.121:41737 #9 (5 connections now open)
 m31002| 2013-10-15T00:12:38.269+0000 [rsSync] replSet initial sync pending
 m31002| 2013-10-15T00:12:38.269+0000 [rsSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:38.270+0000 [initandlisten] connection accepted from 10.159.32.121:41738 #10 (6 connections now open)
 m31002| 2013-10-15T00:12:38.271+0000 [rsSync] build index on: local.replset.minvalid properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "local.replset.minvalid", "name" : "_id_" }
 m31002| 2013-10-15T00:12:38.271+0000 [rsSync] build index done.  scanned 0 total records. 0 secs
 m31002| 2013-10-15T00:12:38.271+0000 [rsSync] replSet initial sync drop all databases
 m31002| 2013-10-15T00:12:38.271+0000 [rsSync] dropAllDatabasesExceptLocal 1
 m31002| 2013-10-15T00:12:38.271+0000 [rsSync] replSet initial sync clone all databases
 m31002| 2013-10-15T00:12:38.272+0000 [rsSync] replSet initial sync cloning db: foo
 m31000| 2013-10-15T00:12:38.272+0000 [initandlisten] connection accepted from 10.159.32.121:41739 #11 (7 connections now open)
 m31002| 2013-10-15T00:12:38.273+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/foo.ns, filling with zeroes...
 m31002| 2013-10-15T00:12:38.292+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/foo.ns, size: 16MB,  took 0.019 secs
 m31002| 2013-10-15T00:12:38.294+0000 [FileAllocator] allocating new datafile /data/db/jstests_initsync1/foo.0, filling with zeroes...
 m31002| 2013-10-15T00:12:38.302+0000 [FileAllocator] done allocating datafile /data/db/jstests_initsync1/foo.0, size: 16MB,  took 0.007 secs
 m31002| 2013-10-15T00:12:38.302+0000 [rsSync] datafileheader::init initializing /data/db/jstests_initsync1/foo.0 n:0
 m31000| 2013-10-15T00:12:38.303+0000 [conn11] exhaust=true sending more
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:38Z"),
    "myState" : 5,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 14,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:38Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:37Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 14,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:38Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:36Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 5,
            "stateStr" : "STARTUP2",
            "uptime" : 17,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "errmsg" : "initial sync cloning db: foo",
            "self" : true
        }
    ],
    "ok" : 1
}
 m31002| 2013-10-15T00:12:38.538+0000 [rsSync] build index on: foo.bar properties: { "v" : 1, "key" : { "_id" : 1 }, "ns" : "foo.bar", "name" : "_id_" }
 m31002| 2013-10-15T00:12:38.539+0000 [rsSync] have free list for foo.$freelist
 m31002| 2013-10-15T00:12:38.648+0000 [rsSync]   fastBuildIndex dupsToDrop:0
 m31002| 2013-10-15T00:12:38.649+0000 [rsSync] build index done.  scanned 10000 total records. 0.11 secs
 m31002| 2013-10-15T00:12:38.649+0000 [rsSync] replSet initial sync cloning db: admin
 m31002| 2013-10-15T00:12:38.649+0000 [rsSync] replSet initial sync data copy, starting syncup
 m31002| 2013-10-15T00:12:38.649+0000 [rsSync] oplog sync 1 of 3
 m31002| 2013-10-15T00:12:38.650+0000 [rsSync] oplog sync 2 of 3
 m31002| 2013-10-15T00:12:38.650+0000 [rsSync] replSet initial sync building indexes
 m31002| 2013-10-15T00:12:38.650+0000 [rsSync] replSet initial sync cloning indexes for : foo
 m31002| 2013-10-15T00:12:38.650+0000 [rsSync] replSet initial sync cloning indexes for : admin
 m31002| 2013-10-15T00:12:38.651+0000 [rsSync] oplog sync 3 of 3
 m31000| 2013-10-15T00:12:38.651+0000 [conn11] end connection 10.159.32.121:41739 (6 connections now open)
 m31002| 2013-10-15T00:12:38.651+0000 [rsSync] replSet initial sync finishing up
 m31002| 2013-10-15T00:12:38.693+0000 [rsSync] replSet set minValid=525c8865:1
 m31002| 2013-10-15T00:12:38.707+0000 [rsSync] replSet RECOVERING
 m31002| 2013-10-15T00:12:38.707+0000 [rsSync] replSet initial sync done
 m31000| 2013-10-15T00:12:38.707+0000 [conn10] end connection 10.159.32.121:41738 (5 connections now open)
 m31001| 2013-10-15T00:12:38.805+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state RECOVERING
 m31002| 2013-10-15T00:12:39.211+0000 [rsBackgroundSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:39.212+0000 [initandlisten] connection accepted from 10.159.32.121:41742 #12 (6 connections now open)
 m31002| 2013-10-15T00:12:39.212+0000 [rsBackgroundSync] replSet remoteOldestOp:    Oct 15 00:11:57 525c884d:b
 m31002| 2013-10-15T00:12:39.212+0000 [rsBackgroundSync] replSet lastOpTimeFetched: Oct 15 00:12:21 525c8865:1
 m31000| 2013-10-15T00:12:39.213+0000 [initandlisten] connection accepted from 10.159.32.121:41743 #13 (7 connections now open)
 m31000| 2013-10-15T00:12:39.303+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state RECOVERING
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:39Z"),
    "myState" : 3,
    "syncingTo" : "ip-10-159-32-121:31000",
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 15,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:38Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:39Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 15,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:38Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:38Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 3,
            "stateStr" : "RECOVERING",
            "uptime" : 18,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "errmsg" : "syncing to: ip-10-159-32-121:31000",
            "self" : true
        }
    ],
    "ok" : 1
}
7. Kill #2 in the middle of syncing
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
ReplSetTest stop *** Shutting down mongod in port 31001 ***
2013-10-14 20:12:40 EDT 
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] now exiting
 m31001| dbexit: 2013-10-15T00:12:39.394+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] closing listening socket: 11
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] closing listening socket: 12
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] closing listening socket: 15
 m31002| 2013-10-15T00:12:39.394+0000 [conn5] end connection 10.159.32.121:36447 (2 connections now open)
 m31000| 2013-10-15T00:12:39.395+0000 [conn8] end connection 10.159.32.121:41728 (6 connections now open)
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31001| 2013-10-15T00:12:39.394+0000 [signalProcessingThread] shutdown: closing all files...
 m31001| 2013-10-15T00:12:39.394+0000 [conn4] end connection 10.159.32.121:48800 (2 connections now open)
 m31001| 2013-10-15T00:12:39.395+0000 [conn1] end connection 127.0.0.1:38864 (2 connections now open)
 m31001| 2013-10-15T00:12:39.395+0000 [conn3] end connection 10.159.32.121:48791 (2 connections now open)
 m31000| 2013-10-15T00:12:39.395+0000 [conn6] end connection 10.159.32.121:41704 (5 connections now open)
 m31001| 2013-10-15T00:12:39.398+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:39.398+0000 [signalProcessingThread] closeAllFiles() finished
 m31001| 2013-10-15T00:12:39.398+0000 [signalProcessingThread] shutdown: removing fs lock...
 m31002| 2013-10-15T00:12:39.707+0000 [rsSync] replSet SECONDARY
 m31002| 2013-10-15T00:12:40.212+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31002| 2013-10-15T00:12:40.212+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-159-32-121:31001 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-159-32-121:31002" }
 m31002| 2013-10-15T00:12:40.212+0000 [rsHealthPoll] replset info ip-10-159-32-121:31001 heartbeat failed, retrying
 m31002| 2013-10-15T00:12:40.214+0000 [rsHealthPoll] warning: Failed to connect to 10.159.32.121, reason: errno:111 Connection refused
 m31002| 2013-10-15T00:12:40.214+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-15T00:12:40.214+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31002| 2013-10-15T00:12:40.214+0000 [rsHealthPoll] replSet info ip-10-159-32-121:31001 is down (or slow to respond): 
 m31002| 2013-10-15T00:12:40.214+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state DOWN
2013-10-15T00:12:40.394+0000 shell: stopped mongo program on port 31001
8. Eventually it should become a secondary
if initial sync has started, this will cause it to fail and sleep for 5 minutes
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:40Z"),
    "myState" : 2,
    "syncingTo" : "ip-10-159-32-121:31000",
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 16,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:40Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:39Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:40Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:38Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 19,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "self" : true
        }
    ],
    "ok" : 1
}
9. Bring #2 back up
ReplSetTest n is : 1
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
{
    "useHostName" : true,
    "oplogSize" : 40,
    "keyFile" : undefined,
    "port" : 31001,
    "noprealloc" : "",
    "smallfiles" : "",
    "rest" : "",
    "replSet" : "jstests_initsync1",
    "dbpath" : "$set-$node",
    "restart" : true,
    "pathOpts" : {
        "node" : 1,
        "set" : "jstests_initsync1"
    }
}
ReplSetTest (Re)Starting....
2013-10-15T00:12:40.407+0000 shell: started program /data/mci/git@github.commongodb/mongo.git/master/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet jstests_initsync1 --dbpath /data/db/jstests_initsync1-1 --setParameter enableTestCommands=1
2013-10-15T00:12:40.408+0000 warning: Failed to connect to 127.0.0.1, reason: errno:111 Connection refused
 m31001| 2013-10-15T00:12:40.454+0000 ** WARNING: --rest is specified without --httpinterface,
 m31001| 2013-10-15T00:12:40.454+0000 **          enabling http interface
 m31001| note: noprealloc may hurt performance in many applications
 m31001| 2013-10-15T00:12:40.461+0000 [initandlisten] MongoDB starting : pid=5919 port=31001 dbpath=/data/db/jstests_initsync1-1 64-bit host=ip-10-159-32-121
 m31001| 2013-10-15T00:12:40.461+0000 [initandlisten] _DEBUG build (which is slower)
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] 
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] **       Not recommended for production.
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] 
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] db version v2.5.3-pre-
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] git version: d1c2ae72c4829ed77cb6b7fe42e486910483caff
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 BOOST_LIB_VERSION=1_49
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] allocator: tcmalloc
 m31001| 2013-10-15T00:12:40.462+0000 [initandlisten] options: { dbpath: "/data/db/jstests_initsync1-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "jstests_initsync1", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31001| 2013-10-15T00:12:40.492+0000 [initandlisten] opening db:  local
 m31001| 2013-10-15T00:12:40.494+0000 [initandlisten] opening db:  foo
 m31001| 2013-10-15T00:12:40.497+0000 [initandlisten] _DEBUG ReadContext db wasn't open, will try to open admin.system.roles
 m31001| 2013-10-15T00:12:40.497+0000 [initandlisten] opening db:  admin
 m31001| 2013-10-15T00:12:40.498+0000 [IndexRebuilder] _DEBUG ReadContext db wasn't open, will try to open local.system.namespaces
 m31001| 2013-10-15T00:12:40.498+0000 [IndexRebuilder] opening db:  local
 m31001| 2013-10-15T00:12:40.499+0000 [initandlisten] 
 m31001| debug have W lock but w would suffice for command create
 m31001| 2013-10-15T00:12:40.499+0000 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
 m31001| 2013-10-15T00:12:40.500+0000 [initandlisten] waiting for connections on port 31001
 m31001| 2013-10-15T00:12:40.500+0000 [websvr] admin web console waiting for connections on port 32001
 m31001| 2013-10-15T00:12:40.500+0000 [IndexRebuilder] _DEBUG ReadContext db wasn't open, will try to open foo.system.namespaces
 m31001| 2013-10-15T00:12:40.500+0000 [IndexRebuilder] opening db:  foo
 m31001| 2013-10-15T00:12:40.502+0000 [IndexRebuilder] assertion 17069 not master and slaveOk=false ns:foo.system.namespaces query:{}
 m31001| 2013-10-15T00:12:40.503+0000 [IndexRebuilder] warning: index rebuilding did not complete
 m31001| 2013-10-15T00:12:40.504+0000 [rsStart] replSet I am ip-10-159-32-121:31001
 m31001| 2013-10-15T00:12:40.504+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31002
 m31001| 2013-10-15T00:12:40.504+0000 [rsStart] starting rsHealthPoll for ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:40.504+0000 [rsStart] replSet STARTUP2
 m31001| 2013-10-15T00:12:40.505+0000 [rsSync] replSet SECONDARY
 m31000| 2013-10-15T00:12:40.505+0000 [initandlisten] connection accepted from 10.159.32.121:41748 #14 (6 connections now open)
 m31001| 2013-10-15T00:12:40.505+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is up
 m31001| 2013-10-15T00:12:40.505+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31000 is now in state PRIMARY
 m31001| 2013-10-15T00:12:40.609+0000 [initandlisten] connection accepted from 127.0.0.1:38917 #1 (1 connection now open)
[
    connection to ip-10-159-32-121:31000,
    connection to ip-10-159-32-121:31001
]
 m31000| 2013-10-15T00:12:40.610+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31000| 2013-10-15T00:12:40.611+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-159-32-121:31001 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-159-32-121:31000" }
 m31000| 2013-10-15T00:12:40.611+0000 [rsHealthPoll] replset info ip-10-159-32-121:31001 heartbeat failed, retrying
2013-10-15T00:12:40.611+0000 DBClientCursor::init call() failed
 m31001| 2013-10-15T00:12:40.612+0000 [initandlisten] connection accepted from 10.159.32.121:48825 #2 (2 connections now open)
Error: error doing query: failed
waiting 6
 m31000| 2013-10-15T00:12:41.303+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state SECONDARY
2013-10-15T00:12:41.615+0000 trying reconnect to 127.0.0.1:31001
 m31001| 2013-10-15T00:12:41.615+0000 [initandlisten] connection accepted from 127.0.0.1:38919 #3 (3 connections now open)
2013-10-15T00:12:41.615+0000 reconnect 127.0.0.1:31001 ok
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:41Z"),
    "myState" : 2,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 1,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:40Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:40Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 1,
            "optime" : Timestamp(1381795941, 1),
            "optimeDate" : ISODate("2013-10-15T00:12:21Z"),
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : -1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}
10. Insert some stuff
2013-10-14 20:12:43 EDT 
 m31000| 2013-10-15T00:12:41.627+0000 [conn5] end connection 10.159.32.121:41703 (5 connections now open)
 m31002| 2013-10-15T00:12:42.214+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31002| 2013-10-15T00:12:42.214+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31001| 2013-10-15T00:12:42.217+0000 [initandlisten] connection accepted from 10.159.32.121:48827 #4 (4 connections now open)
 m31002| 2013-10-15T00:12:42.222+0000 [rsHealthPoll] replset info ip-10-159-32-121:31001 thinks that we are down
 m31002| 2013-10-15T00:12:42.222+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is up
 m31002| 2013-10-15T00:12:42.222+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31001 is now in state SECONDARY
 m31002| 2013-10-15T00:12:42.505+0000 [initandlisten] connection accepted from 10.159.32.121:36464 #6 (3 connections now open)
 m31001| 2013-10-15T00:12:42.505+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is up
 m31001| 2013-10-15T00:12:42.505+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state SECONDARY
11. Everyone happy eventually
ReplSetTest awaitReplication: starting: timestamp for primary, ip-10-159-32-121:31000, is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795941, 1) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795941, 1),  "h" : NumberLong("-6786426008776871186"),  "v" : 2,  "op" : "n",  "ns" : "",  "o" : {  "msg" : "Reconfig set",  "version" : 2 } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
 m31001| 2013-10-15T00:12:47.505+0000 [rsBackgroundSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:47.506+0000 [initandlisten] connection accepted from 10.159.32.121:41760 #15 (6 connections now open)
 m31001| 2013-10-15T00:12:47.506+0000 [rsBackgroundSync] replSet remoteOldestOp:    Oct 15 00:11:57 525c884d:b
 m31001| 2013-10-15T00:12:47.506+0000 [rsBackgroundSync] replSet lastOpTimeFetched: Oct 15 00:12:21 525c8865:1
 m31000| 2013-10-15T00:12:47.507+0000 [initandlisten] connection accepted from 10.159.32.121:41761 #16 (7 connections now open)
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795961, 100) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10102) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795961, 100),  "h" : NumberLong("-2344677019336615432"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c88791b8ae164b6e4c665"),  "date" : ISODate("2013-10-15T00:12:41.633Z"),  "x" : 99,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795961, 100) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 10102) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795961, 100),  "h" : NumberLong("-2344677019336615432"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c88791b8ae164b6e4c665"),  "date" : ISODate("2013-10-15T00:12:41.633Z"),  "x" : 99,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795962, 3701) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 15103) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795962, 3701),  "h" : NumberLong("7239602222960456265"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c887a1b8ae164b6e4d9ee"),  "date" : ISODate("2013-10-15T00:12:42.337Z"),  "x" : 5100,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795962, 3701) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 15103) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795962, 3701),  "h" : NumberLong("7239602222960456265"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c887a1b8ae164b6e4d9ee"),  "date" : ISODate("2013-10-15T00:12:42.337Z"),  "x" : 5100,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: timestamp for secondary #1, ip-10-159-32-121:31001, is Timestamp(1381795962, 3701) but latest is Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: last oplog entry (of 20002) for secondary #1, ip-10-159-32-121:31001, is {  "ts" : Timestamp(1381795962, 3701),  "h" : NumberLong("7239602222960456265"),  "v" : 2,  "op" : "i",  "ns" : "foo.bar",  "o" : {  "_id" : ObjectId("525c887a1b8ae164b6e4d9ee"),  "date" : ISODate("2013-10-15T00:12:42.337Z"),  "x" : 5100,  "str" : "all the talk on the market" } }
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is NOT synced
ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1381795963, 4740)
ReplSetTest awaitReplication: checking secondary #1: ip-10-159-32-121:31001
ReplSetTest awaitReplication: secondary #1, ip-10-159-32-121:31001, is synced
ReplSetTest awaitReplication: finished: all 1 secondaries synced at timestamp Timestamp(1381795963, 4740)
13. Check hbmsg
 m31000| 2013-10-15T00:12:50.270+0000 [conn1] replSet replSetTest command received: { replSetTest: 1.0, sethbmsg: "foo bar baz" }
 m31000| 2013-10-15T00:12:50.270+0000 [conn1] replSet foo bar baz
{
    "set" : "jstests_initsync1",
    "date" : ISODate("2013-10-15T00:12:50Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-10-159-32-121:31000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 53,
            "optime" : Timestamp(1381795963, 4740),
            "optimeDate" : ISODate("2013-10-15T00:12:43Z"),
            "errmsg" : "foo bar baz",
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "ip-10-159-32-121:31001",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 50,
            "optime" : Timestamp(1381795961, 630),
            "optimeDate" : ISODate("2013-10-15T00:12:41Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:48Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:48Z"),
            "pingMs" : 0,
            "lastHeartbeatMessage" : "syncing to: ip-10-159-32-121:31000",
            "syncingTo" : "ip-10-159-32-121:31000"
        },
        {
            "_id" : 2,
            "name" : "ip-10-159-32-121:31002",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 29,
            "optime" : Timestamp(1381795963, 4740),
            "optimeDate" : ISODate("2013-10-15T00:12:43Z"),
            "lastHeartbeat" : ISODate("2013-10-15T00:12:49Z"),
            "lastHeartbeatRecv" : ISODate("2013-10-15T00:12:50Z"),
            "pingMs" : 0,
            "syncingTo" : "ip-10-159-32-121:31000"
        }
    ],
    "ok" : 1
}
 m31001| 2013-10-15T00:12:50.276+0000 [conn4] end connection 10.159.32.121:48827 (3 connections now open)
 m31002| 2013-10-15T00:12:50.275+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31002| 2013-10-15T00:12:50.275+0000 [signalProcessingThread] now exiting
 m31002| dbexit: 2013-10-15T00:12:50.276+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] closing listening socket: 14
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] closing listening socket: 15
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31002.sock
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31002| 2013-10-15T00:12:50.276+0000 [signalProcessingThread] shutdown: closing all files...
 m31002| 2013-10-15T00:12:50.276+0000 [conn6] end connection 10.159.32.121:36464 (2 connections now open)
 m31000| 2013-10-15T00:12:50.276+0000 [conn9] end connection 10.159.32.121:41737 (6 connections now open)
 m31002| 2013-10-15T00:12:50.277+0000 [conn1] end connection 127.0.0.1:59238 (2 connections now open)
 m31002| 2013-10-15T00:12:50.277+0000 [conn4] end connection 10.159.32.121:36444 (0 connections now open)
 m31000| 2013-10-15T00:12:50.277+0000 [conn13] end connection 10.159.32.121:41743 (5 connections now open)
 m31002| 2013-10-15T00:12:50.279+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: ip-10-159-32-121:31000
 m31002| 2013-10-15T00:12:50.280+0000 [signalProcessingThread] closeAllFiles() finished
 m31002| 2013-10-15T00:12:50.280+0000 [signalProcessingThread] shutdown: removing fs lock...
 m31001| 2013-10-15T00:12:50.507+0000 [rsHealthPoll] DBClientCursor::init call() failed
 m31001| 2013-10-15T00:12:50.507+0000 [rsHealthPoll] dev: lastError==0 won't report:DBClientBase::findN: transport error: ip-10-159-32-121:31002 ns: admin.$cmd query: { replSetHeartbeat: "jstests_initsync1", v: 2, pv: 1, checkEmpty: false, from: "ip-10-159-32-121:31001" }
 m31001| 2013-10-15T00:12:50.507+0000 [rsHealthPoll] replset info ip-10-159-32-121:31002 heartbeat failed, retrying
 m31001| 2013-10-15T00:12:50.509+0000 [rsHealthPoll] warning: Failed to connect to 10.159.32.121, reason: errno:111 Connection refused
 m31001| 2013-10-15T00:12:50.509+0000 [rsHealthPoll] unable to set SO_RCVTIMEO
 m31001| 2013-10-15T00:12:50.509+0000 [rsHealthPoll] unable to set SO_SNDTIMEO
 m31001| 2013-10-15T00:12:50.509+0000 [rsHealthPoll] replSet info ip-10-159-32-121:31002 is down (or slow to respond): 
 m31001| 2013-10-15T00:12:50.509+0000 [rsHealthPoll] replSet member ip-10-159-32-121:31002 is now in state DOWN
2013-10-15T00:12:51.275+0000 shell: stopped mongo program on port 31002
ReplSetTest n: 0 ports: [ 31000, 31001 ]    31000 number
ReplSetTest stop *** Shutting down mongod in port 31000 ***
 m31000| 2013-10-15T00:12:51.276+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31000| 2013-10-15T00:12:51.276+0000 [signalProcessingThread] now exiting
 m31000| dbexit: 2013-10-15T00:12:51.276+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31000| 2013-10-15T00:12:51.276+0000 [signalProcessingThread] closing listening socket: 8
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] closing listening socket: 9
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] closing listening socket: 12
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31000.sock
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31000| 2013-10-15T00:12:51.277+0000 [signalProcessingThread] shutdown: closing all files...
 m31001| 2013-10-15T00:12:51.277+0000 [conn2] end connection 10.159.32.121:48825 (2 connections now open)
 m31001| 2013-10-15T00:12:51.277+0000 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:51.277+0000 [rsBackgroundSync] replSet syncing to: ip-10-159-32-121:31000
 m31000| 2013-10-15T00:12:51.278+0000 [conn16] end connection 10.159.32.121:41761 (4 connections now open)
 m31000| 2013-10-15T00:12:51.278+0000 [conn14] end connection 10.159.32.121:41748 (4 connections now open)
 m31000| 2013-10-15T00:12:51.278+0000 [conn1] end connection 127.0.0.1:42610 (3 connections now open)
 m31000| 2013-10-15T00:12:51.282+0000 [signalProcessingThread] closeAllFiles() finished
 m31000| 2013-10-15T00:12:51.282+0000 [signalProcessingThread] shutdown: removing fs lock...
 m31001| 2013-10-15T00:12:51.283+0000 [rsBackgroundSync] Socket recv() errno:104 Connection reset by peer 10.159.32.121:31000
 m31001| 2013-10-15T00:12:51.283+0000 [rsBackgroundSync] SocketException: remote: 10.159.32.121:31000 error: 9001 socket exception [RECV_ERROR] server [10.159.32.121:31000] 
 m31001| 2013-10-15T00:12:51.283+0000 [rsBackgroundSync] DBClientCursor::init call() failed
 m31001| 2013-10-15T00:12:51.283+0000 [rsBackgroundSync] replSet sync source problem: 10276 DBClientBase::findN: transport error: ip-10-159-32-121:31000 ns: local.oplog.rs query: {}
 m31001| 2013-10-15T00:12:51.283+0000 [rsBackgroundSync] replSet syncing to: ip-10-159-32-121:31000
 m31001| 2013-10-15T00:12:51.284+0000 [rsBackgroundSync] warning: Failed to connect to 10.159.32.121, reason: errno:111 Connection refused
 m31001| 2013-10-15T00:12:51.284+0000 [rsBackgroundSync] repl: couldn't connect to server ip-10-159-32-121:31000
2013-10-15T00:12:52.276+0000 shell: stopped mongo program on port 31000
ReplSetTest n: 1 ports: [ 31000, 31001 ]    31001 number
ReplSetTest stop *** Shutting down mongod in port 31001 ***
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] now exiting
 m31001| dbexit: 2013-10-15T00:12:52.277+0000 [signalProcessingThread] shutdown: going to close listening sockets...
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] closing listening socket: 17
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] closing listening socket: 18
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] closing listening socket: 21
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] shutdown: going to flush diaglog...
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] shutdown: going to close sockets...
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
 m31001| 2013-10-15T00:12:52.277+0000 [signalProcessingThread] shutdown: closing all files...
 m31001| 2013-10-15T00:12:52.277+0000 [conn3] end connection 127.0.0.1:38919 (1 connection now open)
 m31001| 2013-10-15T00:12:52.277+0000 [conn1] end connection 127.0.0.1:38917 (1 connection now open)
 m31001| 2013-10-15T00:12:52.279+0000 [signalProcessingThread] closeAllFiles() finished
 m31001| 2013-10-15T00:12:52.279+0000 [signalProcessingThread] shutdown: removing fs lock...
2013-10-15T00:12:53.277+0000 shell: stopped mongo program on port 31001
ReplSetTest stopSet deleting all dbpaths
ReplSetTest stopSet *** Shut down repl set - test worked ****