Wed May 8 02:32:17.060 [conn48] end connection 127.0.0.1:50606 (0 connections now open) Wed May 8 02:32:18.150 [initandlisten] connection accepted from 127.0.0.1:50617 #49 (1 connection now open) Wed May 8 02:32:18.150 [conn49] end connection 127.0.0.1:50617 (0 connections now open) MongoDB shell version: 2.5.0-pre- Wed May 8 02:32:18.618 kern.sched unavailable MONGODB-CR ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testSet", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "testSet" } } ReplSetTest Starting.... Resetting db path '/data/db/testSet-0' Wed May 8 02:32:18.654 shell: started program /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet testSet --dbpath /data/db/testSet-0 --setParameter enableTestCommands=1 m31000| note: noprealloc may hurt performance in many applications m31000| Wed May 8 02:32:18.676 kern.sched unavailable m31000| Wed May 8 02:32:18.679 [initandlisten] MongoDB starting : pid=709 port=31000 dbpath=/data/db/testSet-0 64-bit host=bs-osx-106-x86-64-3.10gen.cc m31000| Wed May 8 02:32:18.679 [initandlisten] m31000| Wed May 8 02:32:18.679 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m31000| Wed May 8 02:32:18.679 [initandlisten] ** Not recommended for production. m31000| Wed May 8 02:32:18.679 [initandlisten] m31000| Wed May 8 02:32:18.679 [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 532 processes, 10240 files. Number of processes should be at least 5120 : 0.5 times number of files. m31000| Wed May 8 02:32:18.679 [initandlisten] m31000| Wed May 8 02:32:18.679 [initandlisten] db version v2.5.0-pre- m31000| Wed May 8 02:32:18.679 [initandlisten] git version: c13d75db43b76a51e96dc61c70fecb75bdaccf70 m31000| Wed May 8 02:32:18.679 [initandlisten] build info: Darwin bs-osx-106-x86-64-3.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31000| Wed May 8 02:32:18.679 [initandlisten] allocator: system m31000| Wed May 8 02:32:18.679 [initandlisten] options: { dbpath: "/data/db/testSet-0", noprealloc: true, oplogSize: 40, port: 31000, replSet: "testSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31000| Wed May 8 02:32:18.682 [FileAllocator] allocating new datafile /data/db/testSet-0/local.ns, filling with zeroes... m31000| Wed May 8 02:32:18.682 [FileAllocator] creating directory /data/db/testSet-0/_tmp m31000| Wed May 8 02:32:18.719 [FileAllocator] done allocating datafile /data/db/testSet-0/local.ns, size: 16MB, took 0.037 secs m31000| Wed May 8 02:32:18.719 [FileAllocator] allocating new datafile /data/db/testSet-0/local.0, filling with zeroes... m31000| Wed May 8 02:32:18.791 [FileAllocator] done allocating datafile /data/db/testSet-0/local.0, size: 16MB, took 0.071 secs m31000| Wed May 8 02:32:18.792 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 111ms m31000| Wed May 8 02:32:18.793 [websvr] admin web console waiting for connections on port 32000 m31000| Wed May 8 02:32:18.793 [initandlisten] waiting for connections on port 31000 m31000| Wed May 8 02:32:18.794 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Wed May 8 02:32:18.794 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Wed May 8 02:32:18.856 [initandlisten] connection accepted from 127.0.0.1:50623 #1 (1 connection now open) [ connection to bs-osx-106-x86-64-3.10gen.cc:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testSet", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "testSet" } } ReplSetTest Starting.... Resetting db path '/data/db/testSet-1' Wed May 8 02:32:18.858 shell: started program /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet testSet --dbpath /data/db/testSet-1 --setParameter enableTestCommands=1 m31001| note: noprealloc may hurt performance in many applications m31001| Wed May 8 02:32:18.881 kern.sched unavailable m31001| Wed May 8 02:32:18.884 [initandlisten] MongoDB starting : pid=710 port=31001 dbpath=/data/db/testSet-1 64-bit host=bs-osx-106-x86-64-3.10gen.cc m31001| Wed May 8 02:32:18.884 [initandlisten] m31001| Wed May 8 02:32:18.884 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m31001| Wed May 8 02:32:18.884 [initandlisten] ** Not recommended for production. m31001| Wed May 8 02:32:18.884 [initandlisten] m31001| Wed May 8 02:32:18.884 [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 532 processes, 10240 files. Number of processes should be at least 5120 : 0.5 times number of files. m31001| Wed May 8 02:32:18.884 [initandlisten] m31001| Wed May 8 02:32:18.884 [initandlisten] db version v2.5.0-pre- m31001| Wed May 8 02:32:18.884 [initandlisten] git version: c13d75db43b76a51e96dc61c70fecb75bdaccf70 m31001| Wed May 8 02:32:18.884 [initandlisten] build info: Darwin bs-osx-106-x86-64-3.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31001| Wed May 8 02:32:18.884 [initandlisten] allocator: system m31001| Wed May 8 02:32:18.884 [initandlisten] options: { dbpath: "/data/db/testSet-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "testSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31001| Wed May 8 02:32:18.885 [FileAllocator] allocating new datafile /data/db/testSet-1/local.ns, filling with zeroes... m31001| Wed May 8 02:32:18.885 [FileAllocator] creating directory /data/db/testSet-1/_tmp m31001| Wed May 8 02:32:18.922 [FileAllocator] done allocating datafile /data/db/testSet-1/local.ns, size: 16MB, took 0.036 secs m31001| Wed May 8 02:32:18.922 [FileAllocator] allocating new datafile /data/db/testSet-1/local.0, filling with zeroes... m31001| Wed May 8 02:32:18.979 [FileAllocator] done allocating datafile /data/db/testSet-1/local.0, size: 16MB, took 0.056 secs m31001| Wed May 8 02:32:18.981 [websvr] admin web console waiting for connections on port 32001 m31001| Wed May 8 02:32:18.981 [initandlisten] waiting for connections on port 31001 m31001| Wed May 8 02:32:18.982 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Wed May 8 02:32:18.982 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Wed May 8 02:32:19.059 [initandlisten] connection accepted from 127.0.0.1:50625 #1 (1 connection now open) [ connection to bs-osx-106-x86-64-3.10gen.cc:31000, connection to bs-osx-106-x86-64-3.10gen.cc:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testSet", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "testSet" } } ReplSetTest Starting.... Resetting db path '/data/db/testSet-2' Wed May 8 02:32:19.061 shell: started program /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/mongod --oplogSize 40 --port 31002 --noprealloc --smallfiles --rest --replSet testSet --dbpath /data/db/testSet-2 --setParameter enableTestCommands=1 m31002| note: noprealloc may hurt performance in many applications m31002| Wed May 8 02:32:19.082 kern.sched unavailable m31002| Wed May 8 02:32:19.085 [initandlisten] MongoDB starting : pid=711 port=31002 dbpath=/data/db/testSet-2 64-bit host=bs-osx-106-x86-64-3.10gen.cc m31002| Wed May 8 02:32:19.085 [initandlisten] m31002| Wed May 8 02:32:19.085 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m31002| Wed May 8 02:32:19.086 [initandlisten] ** Not recommended for production. m31002| Wed May 8 02:32:19.086 [initandlisten] m31002| Wed May 8 02:32:19.086 [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 532 processes, 10240 files. Number of processes should be at least 5120 : 0.5 times number of files. m31002| Wed May 8 02:32:19.086 [initandlisten] m31002| Wed May 8 02:32:19.086 [initandlisten] db version v2.5.0-pre- m31002| Wed May 8 02:32:19.086 [initandlisten] git version: c13d75db43b76a51e96dc61c70fecb75bdaccf70 m31002| Wed May 8 02:32:19.086 [initandlisten] build info: Darwin bs-osx-106-x86-64-3.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31002| Wed May 8 02:32:19.086 [initandlisten] allocator: system m31002| Wed May 8 02:32:19.086 [initandlisten] options: { dbpath: "/data/db/testSet-2", noprealloc: true, oplogSize: 40, port: 31002, replSet: "testSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31002| Wed May 8 02:32:19.087 [FileAllocator] allocating new datafile /data/db/testSet-2/local.ns, filling with zeroes... m31002| Wed May 8 02:32:19.087 [FileAllocator] creating directory /data/db/testSet-2/_tmp m31002| Wed May 8 02:32:19.123 [FileAllocator] done allocating datafile /data/db/testSet-2/local.ns, size: 16MB, took 0.035 secs m31002| Wed May 8 02:32:19.123 [FileAllocator] allocating new datafile /data/db/testSet-2/local.0, filling with zeroes... m31002| Wed May 8 02:32:19.183 [FileAllocator] done allocating datafile /data/db/testSet-2/local.0, size: 16MB, took 0.059 secs m31002| Wed May 8 02:32:19.184 [websvr] admin web console waiting for connections on port 32002 m31002| Wed May 8 02:32:19.185 [initandlisten] waiting for connections on port 31002 m31002| Wed May 8 02:32:19.186 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Wed May 8 02:32:19.186 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Wed May 8 02:32:19.262 [initandlisten] connection accepted from 127.0.0.1:50627 #1 (1 connection now open) [ connection to bs-osx-106-x86-64-3.10gen.cc:31000, connection to bs-osx-106-x86-64-3.10gen.cc:31001, connection to bs-osx-106-x86-64-3.10gen.cc:31002 ] { "replSetInitiate" : { "_id" : "testSet", "members" : [ { "_id" : 0, "host" : "bs-osx-106-x86-64-3.10gen.cc:31000" }, { "_id" : 1, "host" : "bs-osx-106-x86-64-3.10gen.cc:31001", "priority" : 0 }, { "_id" : 2, "host" : "bs-osx-106-x86-64-3.10gen.cc:31002", "priority" : 0 } ] } } m31000| Wed May 8 02:32:19.265 [conn1] replSet replSetInitiate admin command received from client m31000| Wed May 8 02:32:19.266 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Wed May 8 02:32:19.267 [initandlisten] connection accepted from 208.52.185.85:50628 #2 (2 connections now open) m31002| Wed May 8 02:32:19.269 [initandlisten] connection accepted from 208.52.185.85:50629 #2 (2 connections now open) m31000| Wed May 8 02:32:19.270 [conn1] replSet replSetInitiate all members seem up m31000| Wed May 8 02:32:19.270 [conn1] ****** m31000| Wed May 8 02:32:19.270 [conn1] creating replication oplog of size: 40MB... m31000| Wed May 8 02:32:19.270 [FileAllocator] allocating new datafile /data/db/testSet-0/local.1, filling with zeroes... m31000| Wed May 8 02:32:19.506 [FileAllocator] done allocating datafile /data/db/testSet-0/local.1, size: 64MB, took 0.236 secs m31000| Wed May 8 02:32:19.522 [conn1] ****** m31000| Wed May 8 02:32:19.522 [conn1] replSet info saving a newer config version to local.system.replset m31000| Wed May 8 02:32:19.537 [conn1] replSet saveConfigLocally done m31000| Wed May 8 02:32:19.537 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Wed May 8 02:32:19.537 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "testSet", members: [ { _id: 0.0, host: "bs-osx-106-x86-64-3.10gen.cc:31000" }, { _id: 1.0, host: "bs-osx-106-x86-64-3.10gen.cc:31001", priority: 0.0 }, { _id: 2.0, host: "bs-osx-106-x86-64-3.10gen.cc:31002", priority: 0.0 } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:268090 reslen:112 272ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Wed May 8 02:32:19.794 [rsStart] replSet I am bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:19.794 [rsStart] replSet STARTUP2 m31000| Wed May 8 02:32:19.795 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is up m31001| Wed May 8 02:32:19.982 [rsStart] trying to contact bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:19.984 [initandlisten] connection accepted from 208.52.185.85:50630 #2 (2 connections now open) m31001| Wed May 8 02:32:19.985 [rsStart] replSet I am bs-osx-106-x86-64-3.10gen.cc:31001 m31001| Wed May 8 02:32:19.985 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Wed May 8 02:32:19.985 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Wed May 8 02:32:19.991 [rsStart] replSet saveConfigLocally done m31001| Wed May 8 02:32:19.992 [rsStart] replSet STARTUP2 m31001| Wed May 8 02:32:19.992 [rsSync] ****** m31001| Wed May 8 02:32:19.992 [rsSync] creating replication oplog of size: 40MB... m31001| Wed May 8 02:32:19.992 [FileAllocator] allocating new datafile /data/db/testSet-1/local.1, filling with zeroes... m31002| Wed May 8 02:32:20.186 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Wed May 8 02:32:20.236 [FileAllocator] done allocating datafile /data/db/testSet-1/local.1, size: 64MB, took 0.243 secs m31001| Wed May 8 02:32:20.256 [rsSync] ****** m31001| Wed May 8 02:32:20.256 [rsSync] replSet initial sync pending m31001| Wed May 8 02:32:20.256 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Wed May 8 02:32:20.795 [rsSync] replSet SECONDARY m31002| Wed May 8 02:32:21.186 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Wed May 8 02:32:21.795 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is up m31000| Wed May 8 02:32:21.795 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31001 thinks that we are down m31000| Wed May 8 02:32:21.795 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state STARTUP2 m31000| Wed May 8 02:32:21.795 [rsMgr] not electing self, bs-osx-106-x86-64-3.10gen.cc:31001 would veto with 'I don't think bs-osx-106-x86-64-3.10gen.cc:31000 is electable' m31000| Wed May 8 02:32:21.796 [rsMgr] not electing self, bs-osx-106-x86-64-3.10gen.cc:31001 would veto with 'I don't think bs-osx-106-x86-64-3.10gen.cc:31000 is electable' m31001| Wed May 8 02:32:21.986 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is up m31001| Wed May 8 02:32:21.986 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state SECONDARY m31002| Wed May 8 02:32:21.986 [initandlisten] connection accepted from 208.52.185.85:50631 #3 (3 connections now open) m31001| Wed May 8 02:32:21.986 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is up m31002| Wed May 8 02:32:22.186 [rsStart] trying to contact bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:22.188 [initandlisten] connection accepted from 208.52.185.85:50632 #3 (3 connections now open) m31002| Wed May 8 02:32:22.188 [rsStart] trying to contact bs-osx-106-x86-64-3.10gen.cc:31001 m31001| Wed May 8 02:32:22.189 [initandlisten] connection accepted from 208.52.185.85:50633 #3 (3 connections now open) m31002| Wed May 8 02:32:22.190 [rsStart] replSet I am bs-osx-106-x86-64-3.10gen.cc:31002 m31002| Wed May 8 02:32:22.190 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Wed May 8 02:32:22.190 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Wed May 8 02:32:22.196 [rsStart] replSet saveConfigLocally done m31002| Wed May 8 02:32:22.196 [rsStart] replSet STARTUP2 m31002| Wed May 8 02:32:22.196 [rsSync] ****** m31002| Wed May 8 02:32:22.196 [rsSync] creating replication oplog of size: 40MB... m31002| Wed May 8 02:32:22.196 [FileAllocator] allocating new datafile /data/db/testSet-2/local.1, filling with zeroes... m31002| Wed May 8 02:32:22.435 [FileAllocator] done allocating datafile /data/db/testSet-2/local.1, size: 64MB, took 0.239 secs m31002| Wed May 8 02:32:22.455 [rsSync] ****** m31002| Wed May 8 02:32:22.455 [rsSync] replSet initial sync pending m31002| Wed May 8 02:32:22.455 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Wed May 8 02:32:23.796 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31002 thinks that we are down m31000| Wed May 8 02:32:23.796 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state STARTUP2 m31000| Wed May 8 02:32:23.796 [rsMgr] not electing self, bs-osx-106-x86-64-3.10gen.cc:31002 would veto with 'I don't think bs-osx-106-x86-64-3.10gen.cc:31000 is electable' m31001| Wed May 8 02:32:23.987 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31002 thinks that we are down m31001| Wed May 8 02:32:23.987 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state STARTUP2 m31002| Wed May 8 02:32:24.191 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is up m31002| Wed May 8 02:32:24.191 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is up m31002| Wed May 8 02:32:24.191 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state SECONDARY m31002| Wed May 8 02:32:24.191 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state STARTUP2 m31000| Wed May 8 02:32:29.797 [rsMgr] replSet info electSelf 0 m31002| Wed May 8 02:32:29.798 [conn2] replSet RECOVERING m31002| Wed May 8 02:32:29.798 [conn2] replSet info voting yea for bs-osx-106-x86-64-3.10gen.cc:31000 (0) m31001| Wed May 8 02:32:29.798 [conn2] replSet RECOVERING m31001| Wed May 8 02:32:29.798 [conn2] replSet info voting yea for bs-osx-106-x86-64-3.10gen.cc:31000 (0) m31001| Wed May 8 02:32:29.988 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state RECOVERING m31002| Wed May 8 02:32:30.192 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state RECOVERING m31000| Wed May 8 02:32:30.796 [rsMgr] replSet PRIMARY m31000| Wed May 8 02:32:31.798 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state RECOVERING m31000| Wed May 8 02:32:31.798 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state RECOVERING m31001| Wed May 8 02:32:31.988 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state PRIMARY m31002| Wed May 8 02:32:32.192 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state PRIMARY m31001| Wed May 8 02:32:33.798 [conn2] end connection 208.52.185.85:50628 (2 connections now open) m31001| Wed May 8 02:32:33.799 [initandlisten] connection accepted from 208.52.185.85:50634 #4 (3 connections now open) m31000| Wed May 8 02:32:35.989 [conn2] end connection 208.52.185.85:50630 (2 connections now open) m31000| Wed May 8 02:32:35.990 [initandlisten] connection accepted from 208.52.185.85:50635 #4 (3 connections now open) m31001| Wed May 8 02:32:36.256 [rsSync] replSet initial sync pending m31001| Wed May 8 02:32:36.256 [rsSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:36.257 [initandlisten] connection accepted from 208.52.185.85:50636 #5 (4 connections now open) m31001| Wed May 8 02:32:36.259 [rsSync] build index local.me { _id: 1 } m31001| Wed May 8 02:32:36.260 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Wed May 8 02:32:36.260 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Wed May 8 02:32:36.260 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Wed May 8 02:32:36.261 [rsSync] replSet initial sync drop all databases m31001| Wed May 8 02:32:36.261 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Wed May 8 02:32:36.261 [rsSync] replSet initial sync clone all databases m31001| Wed May 8 02:32:36.261 [rsSync] replSet initial sync data copy, starting syncup m31001| Wed May 8 02:32:36.261 [rsSync] oplog sync 1 of 3 m31001| Wed May 8 02:32:36.261 [rsSync] oplog sync 2 of 3 m31001| Wed May 8 02:32:36.261 [rsSync] replSet initial sync building indexes m31001| Wed May 8 02:32:36.261 [rsSync] oplog sync 3 of 3 m31001| Wed May 8 02:32:36.261 [rsSync] replSet initial sync finishing up m31001| Wed May 8 02:32:36.273 [rsSync] replSet set minValid=5189f173:1 m31001| Wed May 8 02:32:36.282 [rsSync] replSet initial sync done m31000| Wed May 8 02:32:36.282 [conn5] end connection 208.52.185.85:50636 (3 connections now open) m31001| Wed May 8 02:32:36.993 [rsBackgroundSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:36.995 [initandlisten] connection accepted from 208.52.185.85:50637 #6 (4 connections now open) m31001| Wed May 8 02:32:37.261 [rsSyncNotifier] replset setting oplog notifier to bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:37.263 [initandlisten] connection accepted from 208.52.185.85:50638 #7 (5 connections now open) m31000| Wed May 8 02:32:38.194 [conn3] end connection 208.52.185.85:50632 (4 connections now open) m31000| Wed May 8 02:32:38.194 [initandlisten] connection accepted from 208.52.185.85:50639 #8 (5 connections now open) m31000| Wed May 8 02:32:38.265 [slaveTracking] build index local.slaves { _id: 1 } m31000| Wed May 8 02:32:38.265 [slaveTracking] build index done. scanned 0 total records. 0 secs m31001| Wed May 8 02:32:38.282 [rsSync] replSet SECONDARY m31002| Wed May 8 02:32:38.456 [rsSync] replSet initial sync pending m31002| Wed May 8 02:32:38.456 [rsSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:38.457 [initandlisten] connection accepted from 208.52.185.85:50640 #9 (6 connections now open) m31002| Wed May 8 02:32:38.458 [rsSync] build index local.me { _id: 1 } m31002| Wed May 8 02:32:38.459 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Wed May 8 02:32:38.460 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Wed May 8 02:32:38.460 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Wed May 8 02:32:38.460 [rsSync] replSet initial sync drop all databases m31002| Wed May 8 02:32:38.460 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Wed May 8 02:32:38.460 [rsSync] replSet initial sync clone all databases m31002| Wed May 8 02:32:38.460 [rsSync] replSet initial sync data copy, starting syncup m31002| Wed May 8 02:32:38.460 [rsSync] oplog sync 1 of 3 m31002| Wed May 8 02:32:38.461 [rsSync] oplog sync 2 of 3 m31002| Wed May 8 02:32:38.461 [rsSync] replSet initial sync building indexes m31002| Wed May 8 02:32:38.461 [rsSync] oplog sync 3 of 3 m31002| Wed May 8 02:32:38.461 [rsSync] replSet initial sync finishing up m31002| Wed May 8 02:32:38.473 [rsSync] replSet set minValid=5189f173:1 m31002| Wed May 8 02:32:38.482 [rsSync] replSet initial sync done m31000| Wed May 8 02:32:38.482 [conn9] end connection 208.52.185.85:50640 (5 connections now open) m31002| Wed May 8 02:32:39.198 [rsBackgroundSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:39.199 [initandlisten] connection accepted from 208.52.185.85:50641 #10 (6 connections now open) m31002| Wed May 8 02:32:39.461 [rsSyncNotifier] replset setting oplog notifier to bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:32:39.462 [initandlisten] connection accepted from 208.52.185.85:50642 #11 (7 connections now open) m31000| Wed May 8 02:32:39.800 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state SECONDARY m31002| Wed May 8 02:32:40.194 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31001 is now in state SECONDARY m31002| Wed May 8 02:32:40.483 [rsSync] replSet SECONDARY m31000| Wed May 8 02:32:40.647 [FileAllocator] allocating new datafile /data/db/testSet-0/foo.ns, filling with zeroes... m31000| Wed May 8 02:32:40.684 [FileAllocator] done allocating datafile /data/db/testSet-0/foo.ns, size: 16MB, took 0.036 secs m31000| Wed May 8 02:32:40.685 [FileAllocator] allocating new datafile /data/db/testSet-0/foo.0, filling with zeroes... m31000| Wed May 8 02:32:40.740 [FileAllocator] done allocating datafile /data/db/testSet-0/foo.0, size: 16MB, took 0.055 secs m31000| Wed May 8 02:32:40.742 [conn1] build index foo.bar { _id: 1 } m31000| Wed May 8 02:32:40.743 [conn1] build index done. scanned 0 total records. 0 secs m31002| Wed May 8 02:32:40.743 [FileAllocator] allocating new datafile /data/db/testSet-2/foo.ns, filling with zeroes... m31001| Wed May 8 02:32:40.743 [FileAllocator] allocating new datafile /data/db/testSet-1/foo.ns, filling with zeroes... ReplSetTest awaitReplication: starting: timestamp for primary, bs-osx-106-x86-64-3.10gen.cc:31000, is Timestamp(1367994760, 1) ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1367994760, 1) ReplSetTest awaitReplication: checking secondary #1: bs-osx-106-x86-64-3.10gen.cc:31001 m31001| Wed May 8 02:32:40.860 [FileAllocator] done allocating datafile /data/db/testSet-1/foo.ns, size: 16MB, took 0.111 secs m31001| Wed May 8 02:32:40.861 [FileAllocator] allocating new datafile /data/db/testSet-1/foo.0, filling with zeroes... m31002| Wed May 8 02:32:40.869 [FileAllocator] done allocating datafile /data/db/testSet-2/foo.ns, size: 16MB, took 0.125 secs m31002| Wed May 8 02:32:40.869 [FileAllocator] allocating new datafile /data/db/testSet-2/foo.0, filling with zeroes... m31001| Wed May 8 02:32:40.981 [FileAllocator] done allocating datafile /data/db/testSet-1/foo.0, size: 16MB, took 0.12 secs m31001| Wed May 8 02:32:40.983 [repl writer worker 1] build index foo.bar { _id: 1 } m31002| Wed May 8 02:32:40.989 [FileAllocator] done allocating datafile /data/db/testSet-2/foo.0, size: 16MB, took 0.119 secs m31002| Wed May 8 02:32:40.991 [repl writer worker 1] build index foo.bar { _id: 1 } m31002| Wed May 8 02:32:40.992 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Wed May 8 02:32:40.998 [repl writer worker 1] build index done. scanned 0 total records. 0.014 secs ReplSetTest awaitReplication: secondary #1, bs-osx-106-x86-64-3.10gen.cc:31001, is synced ReplSetTest awaitReplication: checking secondary #2: bs-osx-106-x86-64-3.10gen.cc:31002 ReplSetTest awaitReplication: secondary #2, bs-osx-106-x86-64-3.10gen.cc:31002, is synced ReplSetTest awaitReplication: finished: all 2 secondaries synced at timestamp Timestamp(1367994760, 1) Make sure 2 & 3 are syncing from the primary m31001| Wed May 8 02:32:41.000 [conn1] attempting to sync from bs-osx-106-x86-64-3.10gen.cc:31000, but its latest opTime is 1367994739 and ours is 1367994760 so this may not work m31002| Wed May 8 02:32:41.001 [conn1] attempting to sync from bs-osx-106-x86-64-3.10gen.cc:31000, but its latest opTime is 1367994739 and ours is 1367994760 so this may not work Stop 2's replication Do a few writes Make sure 3 is at write #25 m31000| Wed May 8 02:32:41.003 [conn6] end connection 208.52.185.85:50637 (6 connections now open) m31002| Wed May 8 02:32:41.003 [rsBackgroundSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 by request m31000| Wed May 8 02:32:41.004 [conn10] end connection 208.52.185.85:50641 (5 connections now open) { "ts" : Timestamp(1367994761, 25), "h" : NumberLong("1876332432355843264"), "v" : 2, "op" : "i", "ns" : "foo.bar", "o" : { "_id" : ObjectId("5189f1894ff89541211358e7"), "x" : 24 } } { "ts" : Timestamp(1367994761, 2), "h" : NumberLong("-7359474582425234498"), "v" : 2, "op" : "i", "ns" : "foo.bar", "o" : { "_id" : ObjectId("5189f1894ff89541211358d0"), "x" : 1 } } Stop 3's replication m31000| Wed May 8 02:32:41.004 [initandlisten] connection accepted from 208.52.185.85:50643 #12 (6 connections now open) Start 2's replication Do some writes m31001| Wed May 8 02:32:41.005 [rsBackgroundSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31000 by request m31000| Wed May 8 02:32:41.006 [initandlisten] connection accepted from 208.52.185.85:50644 #13 (7 connections now open) Make sure 2 is at write #50 { "ts" : Timestamp(1367994761, 50), "h" : NumberLong("2766321278246693686"), "v" : 2, "op" : "i", "ns" : "foo.bar", "o" : { "_id" : ObjectId("5189f1894ff8954121135900"), "x" : 49 } } { "ts" : Timestamp(1367994761, 1), "h" : NumberLong("-1356905440117484556"), "v" : 2, "op" : "i", "ns" : "foo.bar", "o" : { "_id" : ObjectId("5189f1894ff89541211358cf"), "x" : 0 } } Stop 2's replication Do some writes - 2 & 3 should have up to write #75 in their buffers, but unapplied waiting a bit for the secondaries to get the write m31000| Wed May 8 02:32:41.800 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state SECONDARY m31001| Wed May 8 02:32:41.990 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31002 is now in state SECONDARY m31002| Wed May 8 02:32:49.801 [conn2] end connection 208.52.185.85:50629 (2 connections now open) m31002| Wed May 8 02:32:49.802 [initandlisten] connection accepted from 208.52.185.85:50649 #4 (3 connections now open) m31002| Wed May 8 02:32:49.991 [conn3] end connection 208.52.185.85:50631 (2 connections now open) m31002| Wed May 8 02:32:49.992 [initandlisten] connection accepted from 208.52.185.85:50650 #5 (3 connections now open) m31001| Wed May 8 02:32:52.196 [conn3] end connection 208.52.185.85:50633 (2 connections now open) m31001| Wed May 8 02:32:52.197 [initandlisten] connection accepted from 208.52.185.85:50651 #5 (3 connections now open) Shut down the primary ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** m31000| Wed May 8 02:33:01.348 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends m31000| Wed May 8 02:33:01.348 [signalProcessingThread] now exiting m31000| Wed May 8 02:33:01.348 dbexit: m31000| Wed May 8 02:33:01.348 [signalProcessingThread] shutdown: going to close listening sockets... m31000| Wed May 8 02:33:01.348 [signalProcessingThread] closing listening socket: 14 m31000| Wed May 8 02:33:01.348 [signalProcessingThread] closing listening socket: 15 m31000| Wed May 8 02:33:01.348 [signalProcessingThread] closing listening socket: 16 m31000| Wed May 8 02:33:01.348 [signalProcessingThread] removing socket file: /tmp/mongodb-31000.sock m31000| Wed May 8 02:33:01.349 [signalProcessingThread] shutdown: going to flush diaglog... m31000| Wed May 8 02:33:01.349 [signalProcessingThread] shutdown: going to close sockets... m31000| Wed May 8 02:33:01.349 [signalProcessingThread] shutdown: waiting for fs preallocator... m31000| Wed May 8 02:33:01.349 [signalProcessingThread] shutdown: closing all files... m31000| Wed May 8 02:33:01.349 [conn1] end connection 127.0.0.1:50623 (6 connections now open) m31000| Wed May 8 02:33:01.349 [conn7] end connection 208.52.185.85:50638 (6 connections now open) m31000| Wed May 8 02:33:01.349 [conn4] end connection 208.52.185.85:50635 (6 connections now open) m31000| Wed May 8 02:33:01.349 [conn11] end connection 208.52.185.85:50642 (6 connections now open) m31000| Wed May 8 02:33:01.349 [conn8] end connection 208.52.185.85:50639 (6 connections now open) m31001| Wed May 8 02:33:01.349 [conn4] end connection 208.52.185.85:50634 (2 connections now open) m31001| Wed May 8 02:33:01.349 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:01.349 [conn4] end connection 208.52.185.85:50649 (2 connections now open) m31002| Wed May 8 02:33:01.349 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: bs-osx-106-x86-64-3.10gen.cc:31000 m31000| Wed May 8 02:33:01.360 [signalProcessingThread] closeAllFiles() finished m31000| Wed May 8 02:33:01.360 [signalProcessingThread] shutdown: removing fs lock... m31000| Wed May 8 02:33:01.360 dbexit: really exiting now m31001| Wed May 8 02:33:01.993 [rsHealthPoll] DBClientCursor::init call() failed m31001| Wed May 8 02:33:01.993 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:01.994 [rsHealthPoll] replSet info bs-osx-106-x86-64-3.10gen.cc:31000 is down (or slow to respond): m31001| Wed May 8 02:33:01.994 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state DOWN m31002| Wed May 8 02:33:02.197 [rsHealthPoll] DBClientCursor::init call() failed m31002| Wed May 8 02:33:02.197 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:02.198 [rsHealthPoll] replSet info bs-osx-106-x86-64-3.10gen.cc:31000 is down (or slow to respond): m31002| Wed May 8 02:33:02.198 [rsHealthPoll] replSet member bs-osx-106-x86-64-3.10gen.cc:31000 is now in state DOWN Wed May 8 02:33:02.348 shell: stopped mongo program on port 31000 3 should not attempt to sync from 2, as it cannot clear its buffer m31001| Wed May 8 02:33:03.995 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:04.199 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:05.996 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:05.997 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:05.998 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:05.998 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:05.999 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:06.000 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:06.200 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:08.001 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:08.001 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:08.002 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:08.003 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:08.003 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:08.004 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:08.201 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:08.202 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:08.202 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:08.203 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:08.203 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:08.204 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:10.005 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:10.006 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:10.006 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:10.007 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:10.008 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:10.008 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:10.205 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:10.206 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:10.207 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:10.207 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:10.208 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:10.209 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:12.009 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:12.010 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:12.011 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:12.011 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:12.012 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:12.013 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:12.209 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:12.210 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:12.211 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:12.212 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:12.212 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:12.213 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:14.013 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:14.014 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:14.015 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:14.016 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:14.016 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:14.017 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:14.214 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:14.215 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:14.215 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:14.216 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:14.217 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:14.218 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:16.018 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:16.019 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:16.020 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:16.020 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:16.021 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:16.022 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:16.218 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:16.219 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:16.220 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:16.221 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:16.221 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:16.222 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:18.023 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:18.024 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:18.024 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:18.025 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:18.026 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:18.027 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:18.223 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:18.224 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:18.225 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:18.225 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:18.226 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:18.227 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:19.996 [conn5] end connection 208.52.185.85:50650 (1 connection now open) m31002| Wed May 8 02:33:19.997 [initandlisten] connection accepted from 208.52.185.85:50737 #6 (2 connections now open) m31001| Wed May 8 02:33:20.028 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:20.028 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:20.029 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:20.029 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:20.030 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:20.031 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:20.228 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:20.228 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:20.229 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:20.230 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:20.230 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:20.231 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.032 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.032 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.033 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:22.034 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.034 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.035 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:22.201 [conn5] end connection 208.52.185.85:50651 (1 connection now open) m31001| Wed May 8 02:33:22.201 [initandlisten] connection accepted from 208.52.185.85:50756 #6 (2 connections now open) m31002| Wed May 8 02:33:22.232 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:22.233 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:22.233 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:22.234 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:22.235 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:22.236 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:24.036 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:24.037 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:24.038 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:24.038 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:24.039 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:24.039 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:24.236 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:24.237 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:24.238 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:24.238 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:24.239 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:24.240 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:26.040 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:26.041 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:26.042 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:26.043 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:26.043 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:26.044 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:26.241 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:26.241 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:26.242 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:26.243 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:26.243 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:26.244 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:28.045 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:28.046 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:28.046 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:28.047 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:28.048 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:28.048 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:28.245 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:28.246 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:28.246 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:28.247 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:28.248 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:28.248 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:30.049 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:30.050 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:30.051 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:30.052 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:30.052 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:30.053 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:30.249 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:30.250 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:30.251 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:30.252 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:30.252 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:30.253 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:32.054 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:32.055 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:32.055 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:32.056 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:32.057 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:32.057 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.254 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.255 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.256 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:32.256 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.257 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.258 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 assert.soon failed: function () { var syncingTo = member3.adminCommand({replSetGetStatus:1}).syncingTo; return syncingTo == getHostName()+":"+replSet.ports[1]; }, msg:undefined Error: Printing Stack Trace at printStackTrace (src/mongo/shell/utils.js:37:15) at doassert (src/mongo/shell/assert.js:6:5) at Function.assert.soon (src/mongo/shell/assert.js:110:60) at /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/jstests/replsets/server8070.js:106:20 at Function.assert.throws (src/mongo/shell/assert.js:138:14) at fixed (/data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/jstests/replsets/server8070.js:104:12) at /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/jstests/replsets/server8070.js:117:1 --- pause 3's bgsync thread --- Allow 3 to apply ops 25-75 m31002| Wed May 8 02:33:32.421 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:32.423 [rsBackgroundSync] replSet syncing to: bs-osx-106-x86-64-3.10gen.cc:31001 m31001| Wed May 8 02:33:32.424 [initandlisten] connection accepted from 208.52.185.85:50826 #7 (3 connections now open) primary: 1367994761 secondary: 1367994761 --- start 3's bgsync thread --- Shouldn't hit rollback m31002| Wed May 8 02:33:33.421 [rsSyncNotifier] replset setting oplog notifier to bs-osx-106-x86-64-3.10gen.cc:31001 m31002| Wed May 8 02:33:33.421 [rsSyncNotifier] Socket flush send() errno:32 Broken pipe 208.52.185.85:31000 m31002| Wed May 8 02:33:33.421 [rsSyncNotifier] caught exception (socket exception [SEND_ERROR] for 208.52.185.85:31000) in destructor (~PiggyBackData) m31001| Wed May 8 02:33:33.422 [initandlisten] connection accepted from 208.52.185.85:50828 #8 (4 connections now open) m31001| Wed May 8 02:33:34.058 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:34.059 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:34.060 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:34.060 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:34.061 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:34.062 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:34.259 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:34.260 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:34.260 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:34.261 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:34.262 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:34.262 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.063 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.063 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.064 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.065 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31001| Wed May 8 02:33:36.065 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.066 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31001| Wed May 8 02:33:36.067 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:36.263 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:36.264 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:36.264 [rsHealthPoll] replset info bs-osx-106-x86-64-3.10gen.cc:31000 heartbeat failed, retrying m31002| Wed May 8 02:33:36.265 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:36.266 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:36.266 [rsHealthPoll] couldn't connect to bs-osx-106-x86-64-3.10gen.cc:31000: couldn't connect to server bs-osx-106-x86-64-3.10gen.cc:31000 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet we are ahead of the sync source, will try to roll back m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet rollback 0 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet ROLLBACK m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet rollback 1 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet rollback 2 FindCommonPoint m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet info rollback our last optime: May 8 02:32:41:4b m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet info rollback their last optime: May 8 02:32:41:32 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet info rollback diff in end of log times: 0 seconds m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet rollback found matching events at May 8 02:32:41:32 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet rollback findcommonpoint scanned : 26 m31002| Wed May 8 02:33:37.481 [rsBackgroundSync] replSet replSet rollback 3 fixup m31002| Wed May 8 02:33:37.501 [rsBackgroundSync] replSet rollback 3.5 m31002| Wed May 8 02:33:37.501 [rsBackgroundSync] replSet rollback 4 n:25 m31002| Wed May 8 02:33:37.501 [rsBackgroundSync] replSet minvalid=May 8 02:32:41 5189f189:32 m31002| Wed May 8 02:33:37.501 [rsBackgroundSync] replSet rollback 4.6 m31002| Wed May 8 02:33:37.501 [rsBackgroundSync] replSet rollback 4.7 m31002| Wed May 8 02:33:37.502 [rsBackgroundSync] replSet rollback 5 d:50 u:0 m31002| Wed May 8 02:33:37.518 [rsBackgroundSync] replSet rollback 6 m31002| Wed May 8 02:33:37.518 [rsBackgroundSync] replSet rollback 7 assert failed Error: Printing Stack Trace at printStackTrace (src/mongo/shell/utils.js:37:15) at doassert (src/mongo/shell/assert.js:6:5) at assert (src/mongo/shell/assert.js:14:5) at /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/jstests/replsets/server8070.js:139:5 Wed May 8 02:33:37.524 JavaScript execution failed: assert failed at src/mongo/shell/assert.js:L7 failed to load: /data/buildslaves/OS_X_105_64bit_DUR_OFF/mongo/jstests/replsets/server8070.js Wed May 8 02:33:39.916 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends Wed May 8 02:33:39.916 [signalProcessingThread] now exiting Wed May 8 02:33:39.916 dbexit: Wed May 8 02:33:39.916 [signalProcessingThread] shutdown: going to close listening sockets... Wed May 8 02:33:39.916 [signalProcessingThread] closing listening socket: 11 Wed May 8 02:33:39.916 [signalProcessingThread] closing listening socket: 12 Wed May 8 02:33:39.916 [signalProcessingThread] closing listening socket: 13 Wed May 8 02:33:39.917 [signalProcessingThread] removing socket file: /tmp/mongodb-27999.sock Wed May 8 02:33:39.917 [signalProcessingThread] shutdown: going to flush diaglog... Wed May 8 02:33:39.917 [signalProcessingThread] shutdown: going to close sockets... Wed May 8 02:33:39.917 [signalProcessingThread] shutdown: waiting for fs preallocator... Wed May 8 02:33:39.917 [signalProcessingThread] shutdown: closing all files... Wed May 8 02:33:39.924 [signalProcessingThread] closeAllFiles() finished Wed May 8 02:33:39.924 [signalProcessingThread] shutdown: removing fs lock... Wed May 8 02:33:39.924 dbexit: really exiting now