cwd [/home/gregorv/Workspaces/10Gen/mongo] num procs:221 removing: /data/db/sconsTests//local.ns removing: /data/db/sconsTests//mongod.lock removing: /data/db/sconsTests//local.0 Wed Jan 16 17:18:15.667 [initandlisten] MongoDB starting : pid=3617 port=27999 dbpath=/data/db/sconsTests/ 64-bit host=nuwen Wed Jan 16 17:18:15.667 [initandlisten] Wed Jan 16 17:18:15.667 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. Wed Jan 16 17:18:15.667 [initandlisten] ** Not recommended for production. Wed Jan 16 17:18:15.667 [initandlisten] Wed Jan 16 17:18:15.667 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 Wed Jan 16 17:18:15.667 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 Wed Jan 16 17:18:15.667 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 Wed Jan 16 17:18:15.668 [initandlisten] allocator: tcmalloc Wed Jan 16 17:18:15.668 [initandlisten] options: { dbpath: "/data/db/sconsTests/", nopreallocj: true, port: 27999, setParameter: [ "enableTestCommands=1" ] } Wed Jan 16 17:18:15.729 [initandlisten] journal dir=/data/db/sconsTests/journal Wed Jan 16 17:18:15.729 [initandlisten] recover : no journal files present, no recovery needed Wed Jan 16 17:18:15.780 [FileAllocator] allocating new datafile /data/db/sconsTests/local.ns, filling with zeroes... Wed Jan 16 17:18:15.780 [FileAllocator] creating directory /data/db/sconsTests/_tmp Wed Jan 16 17:18:15.955 [FileAllocator] done allocating datafile /data/db/sconsTests/local.ns, size: 16MB, took 0.141 secs Wed Jan 16 17:18:15.955 [FileAllocator] allocating new datafile /data/db/sconsTests/local.0, filling with zeroes... Wed Jan 16 17:18:16.488 [FileAllocator] done allocating datafile /data/db/sconsTests/local.0, size: 64MB, took 0.532 secs Wed Jan 16 17:18:16.489 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 709ms Wed Jan 16 17:18:16.490 [initandlisten] waiting for connections on port 27999 Wed Jan 16 17:18:16.490 [websvr] admin web console waiting for connections on port 28999 running /home/gregorv/Workspaces/10Gen/mongo/mongod --port 27999 --dbpath /data/db/sconsTests/ --setParameter enableTestCommands=1 --nopreallocj ******************************************* Test : sync_change_source.js ... Command : /home/gregorv/Workspaces/10Gen/mongo/mongo --port 27999 --authenticationMechanism MONGO-CR /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js --eval TestData = new Object();TestData.testPath = "/home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js";TestData.testFile = "sync_change_source.js";TestData.testName = "sync_change_source";TestData.noJournal = false;TestData.noJournalPrealloc = true;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null; Date : Wed Jan 16 17:18:16 2013 MongoDB shell version: 2.3.3-pre- connecting to: 127.0.0.1:27999/test Wed Jan 16 17:18:16.677 [initandlisten] connection accepted from 127.0.0.1:53623 #2 (1 connection now open) null Running: sudo bash -c tc qdisc del root dev lo > /tmp/output Wed Jan 16 17:18:16.682 shell: started program sudo bash -c tc qdisc del root dev lo > /tmp/output Running: sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Wed Jan 16 17:22:36.689 shell: started program sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Running: sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Wed Jan 16 17:22:36.703 shell: started program sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output Wed Jan 16 17:22:36.717 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.2 Resetting db path '/data/db/testReplSet-0' Wed Jan 16 17:22:36.764 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-0 --nopreallocj --bind_ip 127.0.0.2 --setParameter enableTestCommands=1 m31000| note: noprealloc may hurt performance in many applications m31000| Wed Jan 16 17:22:36.788 [initandlisten] MongoDB starting : pid=3669 port=31000 dbpath=/data/db/testReplSet-0 64-bit host=nuwen m31000| Wed Jan 16 17:22:36.788 [initandlisten] m31000| Wed Jan 16 17:22:36.788 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31000| Wed Jan 16 17:22:36.788 [initandlisten] ** Not recommended for production. m31000| Wed Jan 16 17:22:36.788 [initandlisten] m31000| Wed Jan 16 17:22:36.788 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31000| Wed Jan 16 17:22:36.788 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31000| Wed Jan 16 17:22:36.788 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31000| Wed Jan 16 17:22:36.788 [initandlisten] allocator: tcmalloc m31000| Wed Jan 16 17:22:36.788 [initandlisten] options: { bind_ip: "127.0.0.2", dbpath: "/data/db/testReplSet-0", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31000, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31000| Wed Jan 16 17:22:36.855 [initandlisten] journal dir=/data/db/testReplSet-0/journal m31000| Wed Jan 16 17:22:36.855 [initandlisten] recover : no journal files present, no recovery needed m31000| Wed Jan 16 17:22:36.906 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.ns, filling with zeroes... m31000| Wed Jan 16 17:22:36.906 [FileAllocator] creating directory /data/db/testReplSet-0/_tmp m31000| Wed Jan 16 17:22:37.089 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.ns, size: 16MB, took 0.15 secs m31000| Wed Jan 16 17:22:37.089 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.0, filling with zeroes... m31000| Wed Jan 16 17:22:37.230 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.0, size: 16MB, took 0.141 secs m31000| Wed Jan 16 17:22:37.232 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 326ms m31000| Wed Jan 16 17:22:37.232 [initandlisten] waiting for connections on port 31000 m31000| Wed Jan 16 17:22:37.232 [websvr] admin web console waiting for connections on port 32000 m31000| Wed Jan 16 17:22:37.234 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Wed Jan 16 17:22:37.234 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Wed Jan 16 17:22:37.368 [initandlisten] connection accepted from 127.0.0.1:53342 #1 (1 connection now open) [ connection to 127.0.0.2:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.3 Resetting db path '/data/db/testReplSet-1' Wed Jan 16 17:22:37.399 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-1 --nopreallocj --bind_ip 127.0.0.3 --setParameter enableTestCommands=1 m31001| note: noprealloc may hurt performance in many applications m31001| Wed Jan 16 17:22:37.423 [initandlisten] MongoDB starting : pid=3720 port=31001 dbpath=/data/db/testReplSet-1 64-bit host=nuwen m31001| Wed Jan 16 17:22:37.423 [initandlisten] m31001| Wed Jan 16 17:22:37.423 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31001| Wed Jan 16 17:22:37.423 [initandlisten] ** Not recommended for production. m31001| Wed Jan 16 17:22:37.423 [initandlisten] m31001| Wed Jan 16 17:22:37.423 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31001| Wed Jan 16 17:22:37.423 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31001| Wed Jan 16 17:22:37.423 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31001| Wed Jan 16 17:22:37.423 [initandlisten] allocator: tcmalloc m31001| Wed Jan 16 17:22:37.423 [initandlisten] options: { bind_ip: "127.0.0.3", dbpath: "/data/db/testReplSet-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31001, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31001| Wed Jan 16 17:22:37.473 [initandlisten] journal dir=/data/db/testReplSet-1/journal m31001| Wed Jan 16 17:22:37.473 [initandlisten] recover : no journal files present, no recovery needed m31001| Wed Jan 16 17:22:37.524 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.ns, filling with zeroes... m31001| Wed Jan 16 17:22:37.524 [FileAllocator] creating directory /data/db/testReplSet-1/_tmp m31001| Wed Jan 16 17:22:37.707 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.ns, size: 16MB, took 0.15 secs m31001| Wed Jan 16 17:22:37.707 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.0, filling with zeroes... m31001| Wed Jan 16 17:22:37.848 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.0, size: 16MB, took 0.141 secs m31001| Wed Jan 16 17:22:37.849 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 325ms m31001| Wed Jan 16 17:22:37.850 [initandlisten] waiting for connections on port 31001 m31001| Wed Jan 16 17:22:37.850 [websvr] admin web console waiting for connections on port 32001 m31001| Wed Jan 16 17:22:37.851 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Wed Jan 16 17:22:37.851 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Wed Jan 16 17:22:38.000 [initandlisten] connection accepted from 127.0.0.1:56462 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.4 Resetting db path '/data/db/testReplSet-2' Wed Jan 16 17:22:38.028 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31002 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-2 --nopreallocj --bind_ip 127.0.0.4 --setParameter enableTestCommands=1 m31002| note: noprealloc may hurt performance in many applications m31002| Wed Jan 16 17:22:38.054 [initandlisten] MongoDB starting : pid=3772 port=31002 dbpath=/data/db/testReplSet-2 64-bit host=nuwen m31002| Wed Jan 16 17:22:38.054 [initandlisten] m31002| Wed Jan 16 17:22:38.054 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31002| Wed Jan 16 17:22:38.054 [initandlisten] ** Not recommended for production. m31002| Wed Jan 16 17:22:38.054 [initandlisten] m31002| Wed Jan 16 17:22:38.054 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31002| Wed Jan 16 17:22:38.054 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31002| Wed Jan 16 17:22:38.054 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31002| Wed Jan 16 17:22:38.054 [initandlisten] allocator: tcmalloc m31002| Wed Jan 16 17:22:38.054 [initandlisten] options: { bind_ip: "127.0.0.4", dbpath: "/data/db/testReplSet-2", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31002, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31002| Wed Jan 16 17:22:38.116 [initandlisten] journal dir=/data/db/testReplSet-2/journal m31002| Wed Jan 16 17:22:38.116 [initandlisten] recover : no journal files present, no recovery needed m31002| Wed Jan 16 17:22:38.159 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.ns, filling with zeroes... m31002| Wed Jan 16 17:22:38.160 [FileAllocator] creating directory /data/db/testReplSet-2/_tmp m31002| Wed Jan 16 17:22:38.343 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.ns, size: 16MB, took 0.15 secs m31002| Wed Jan 16 17:22:38.343 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.0, filling with zeroes... m31002| Wed Jan 16 17:22:38.484 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.0, size: 16MB, took 0.141 secs m31002| Wed Jan 16 17:22:38.485 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 326ms m31002| Wed Jan 16 17:22:38.486 [initandlisten] waiting for connections on port 31002 m31002| Wed Jan 16 17:22:38.486 [websvr] admin web console waiting for connections on port 32002 m31002| Wed Jan 16 17:22:38.488 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Wed Jan 16 17:22:38.488 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Wed Jan 16 17:22:38.629 [initandlisten] connection accepted from 127.0.0.1:57362 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002 ] { "replSetInitiate" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000" }, { "_id" : 1, "host" : "127.0.0.3:31001" }, { "_id" : 2, "host" : "127.0.0.4:31002" } ] } } m31000| Wed Jan 16 17:22:38.633 [conn1] replSet replSetInitiate admin command received from client m31000| Wed Jan 16 17:22:38.633 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Wed Jan 16 17:22:38.633 [initandlisten] connection accepted from 127.0.0.2:59080 #2 (2 connections now open) m31002| Wed Jan 16 17:22:38.634 [initandlisten] connection accepted from 127.0.0.2:60525 #2 (2 connections now open) m31000| Wed Jan 16 17:22:38.635 [conn1] replSet replSetInitiate all members seem up m31000| Wed Jan 16 17:22:38.635 [conn1] ****** m31000| Wed Jan 16 17:22:38.635 [conn1] creating replication oplog of size: 40MB... m31000| Wed Jan 16 17:22:38.635 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.1, filling with zeroes... m31000| Wed Jan 16 17:22:39.151 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.1, size: 64MB, took 0.515 secs m31000| Wed Jan 16 17:22:40.250 [conn1] ****** m31000| Wed Jan 16 17:22:40.250 [conn1] replSet info saving a newer config version to local.system.replset m31000| Wed Jan 16 17:22:40.252 [conn1] replSet saveConfigLocally done m31000| Wed Jan 16 17:22:40.252 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Wed Jan 16 17:22:40.253 [conn1] build index local.replset.minvalid { _id: 1 } m31000| Wed Jan 16 17:22:40.253 [conn1] build index done. scanned 0 total records. 0 secs m31000| Wed Jan 16 17:22:40.253 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000" }, { _id: 1.0, host: "127.0.0.3:31001" }, { _id: 2.0, host: "127.0.0.4:31002" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1618642 reslen:112 1620ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Wed Jan 16 17:22:47.235 [rsStart] replSet I am 127.0.0.2:31000 m31000| Wed Jan 16 17:22:47.235 [rsStart] replSet STARTUP2 m31000| Wed Jan 16 17:22:47.235 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Wed Jan 16 17:22:47.235 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Wed Jan 16 17:22:47.852 [rsStart] trying to contact 127.0.0.2:31000 m31000| Wed Jan 16 17:22:47.852 [initandlisten] connection accepted from 127.0.0.3:58847 #2 (2 connections now open) m31001| Wed Jan 16 17:22:47.853 [rsStart] replSet I am 127.0.0.3:31001 m31001| Wed Jan 16 17:22:47.853 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Wed Jan 16 17:22:47.853 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Wed Jan 16 17:22:48.154 [rsStart] replSet saveConfigLocally done m31001| Wed Jan 16 17:22:48.154 [rsStart] replSet STARTUP2 m31001| Wed Jan 16 17:22:48.155 [rsSync] ****** m31001| Wed Jan 16 17:22:48.155 [rsSync] creating replication oplog of size: 40MB... m31000| Wed Jan 16 17:22:48.235 [rsSync] replSet SECONDARY m31001| Wed Jan 16 17:22:48.243 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.1, filling with zeroes... m31002| Wed Jan 16 17:22:48.488 [rsStart] trying to contact 127.0.0.2:31000 m31000| Wed Jan 16 17:22:48.489 [initandlisten] connection accepted from 127.0.0.4:35732 #3 (3 connections now open) m31002| Wed Jan 16 17:22:48.489 [rsStart] replSet I am 127.0.0.4:31002 m31002| Wed Jan 16 17:22:48.489 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Wed Jan 16 17:22:48.489 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Wed Jan 16 17:22:48.706 [rsStart] replSet saveConfigLocally done m31002| Wed Jan 16 17:22:48.706 [rsStart] replSet STARTUP2 m31002| Wed Jan 16 17:22:48.707 [rsSync] ****** m31002| Wed Jan 16 17:22:48.707 [rsSync] creating replication oplog of size: 40MB... m31002| Wed Jan 16 17:22:48.748 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.1, filling with zeroes... m31001| Wed Jan 16 17:22:48.781 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.1, size: 64MB, took 0.537 secs m31000| Wed Jan 16 17:22:49.235 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:22:49.235 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31000| Wed Jan 16 17:22:49.236 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31000| Wed Jan 16 17:22:49.236 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31000| Wed Jan 16 17:22:49.236 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31000| Wed Jan 16 17:22:49.236 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31002| Wed Jan 16 17:22:49.691 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.1, size: 64MB, took 0.943 secs m31001| Wed Jan 16 17:22:49.691 [rsSync] ****** m31001| Wed Jan 16 17:22:49.691 [rsSync] replSet initial sync pending m31001| Wed Jan 16 17:22:49.691 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31001| Wed Jan 16 17:22:49.854 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Wed Jan 16 17:22:49.854 [initandlisten] connection accepted from 127.0.0.3:38174 #3 (3 connections now open) m31001| Wed Jan 16 17:22:49.854 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Wed Jan 16 17:22:49.854 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31001| Wed Jan 16 17:22:49.854 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Wed Jan 16 17:22:49.854 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31001| Wed Jan 16 17:22:50.490 [initandlisten] connection accepted from 127.0.0.4:34256 #3 (3 connections now open) m31002| Wed Jan 16 17:22:50.490 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Wed Jan 16 17:22:50.490 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Wed Jan 16 17:22:50.490 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Wed Jan 16 17:22:50.490 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31002| Wed Jan 16 17:22:50.590 [rsSync] ****** m31002| Wed Jan 16 17:22:50.590 [rsSync] replSet initial sync pending m31002| Wed Jan 16 17:22:50.590 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Wed Jan 16 17:22:55.237 [rsMgr] replSet info electSelf 0 m31002| Wed Jan 16 17:22:55.237 [conn2] replSet RECOVERING m31002| Wed Jan 16 17:22:55.237 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Wed Jan 16 17:22:55.237 [conn2] replSet RECOVERING m31001| Wed Jan 16 17:22:55.237 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Wed Jan 16 17:22:55.855 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31000| Wed Jan 16 17:22:56.236 [rsMgr] replSet PRIMARY m31002| Wed Jan 16 17:22:56.491 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Wed Jan 16 17:22:56.491 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31000| Wed Jan 16 17:22:57.237 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31000| Wed Jan 16 17:22:57.237 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31001| Wed Jan 16 17:22:57.855 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Wed Jan 16 17:23:01.237 [conn2] end connection 127.0.0.2:59080 (2 connections now open) m31001| Wed Jan 16 17:23:01.238 [initandlisten] connection accepted from 127.0.0.2:51015 #4 (3 connections now open) m31000| Wed Jan 16 17:23:03.857 [conn2] end connection 127.0.0.3:58847 (2 connections now open) m31000| Wed Jan 16 17:23:03.857 [initandlisten] connection accepted from 127.0.0.3:43374 #4 (3 connections now open) m31000| Wed Jan 16 17:23:04.492 [conn3] end connection 127.0.0.4:35732 (2 connections now open) m31000| Wed Jan 16 17:23:04.492 [initandlisten] connection accepted from 127.0.0.4:53839 #5 (3 connections now open) m31001| Wed Jan 16 17:23:05.692 [rsSync] replSet initial sync pending m31001| Wed Jan 16 17:23:05.692 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Wed Jan 16 17:23:05.692 [initandlisten] connection accepted from 127.0.0.3:45872 #6 (4 connections now open) m31001| Wed Jan 16 17:23:05.693 [rsSync] build index local.me { _id: 1 } m31001| Wed Jan 16 17:23:05.694 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Wed Jan 16 17:23:05.694 [rsSync] replSet initial sync drop all databases m31001| Wed Jan 16 17:23:05.694 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Wed Jan 16 17:23:05.694 [rsSync] replSet initial sync clone all databases m31001| Wed Jan 16 17:23:05.695 [rsSync] replSet initial sync data copy, starting syncup m31001| Wed Jan 16 17:23:05.695 [rsSync] oplog sync 1 of 3 m31001| Wed Jan 16 17:23:05.695 [rsSync] oplog sync 2 of 3 m31001| Wed Jan 16 17:23:05.695 [rsSync] replSet initial sync building indexes m31001| Wed Jan 16 17:23:05.695 [rsSync] oplog sync 3 of 3 m31001| Wed Jan 16 17:23:05.696 [rsSync] replSet initial sync finishing up m31001| Wed Jan 16 17:23:05.696 [rsSync] replSet set minValid=50f72830:1 m31001| Wed Jan 16 17:23:05.696 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Wed Jan 16 17:23:05.697 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Wed Jan 16 17:23:05.697 [rsSync] replSet initial sync done m31000| Wed Jan 16 17:23:05.698 [conn6] end connection 127.0.0.3:45872 (3 connections now open) m31001| Wed Jan 16 17:23:06.156 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Wed Jan 16 17:23:06.156 [initandlisten] connection accepted from 127.0.0.3:38016 #7 (4 connections now open) m31002| Wed Jan 16 17:23:06.590 [rsSync] replSet initial sync pending m31002| Wed Jan 16 17:23:06.590 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Wed Jan 16 17:23:06.591 [initandlisten] connection accepted from 127.0.0.4:40081 #8 (5 connections now open) m31002| Wed Jan 16 17:23:06.592 [rsSync] build index local.me { _id: 1 } m31002| Wed Jan 16 17:23:06.592 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Wed Jan 16 17:23:06.593 [rsSync] replSet initial sync drop all databases m31002| Wed Jan 16 17:23:06.593 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Wed Jan 16 17:23:06.593 [rsSync] replSet initial sync clone all databases m31002| Wed Jan 16 17:23:06.593 [rsSync] replSet initial sync data copy, starting syncup m31002| Wed Jan 16 17:23:06.593 [rsSync] oplog sync 1 of 3 m31002| Wed Jan 16 17:23:06.594 [rsSync] oplog sync 2 of 3 m31002| Wed Jan 16 17:23:06.594 [rsSync] replSet initial sync building indexes m31002| Wed Jan 16 17:23:06.594 [rsSync] oplog sync 3 of 3 m31002| Wed Jan 16 17:23:06.594 [rsSync] replSet initial sync finishing up m31002| Wed Jan 16 17:23:06.595 [rsSync] replSet set minValid=50f72830:1 m31002| Wed Jan 16 17:23:06.595 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Wed Jan 16 17:23:06.595 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Wed Jan 16 17:23:06.596 [rsSync] replSet initial sync done m31000| Wed Jan 16 17:23:06.596 [conn8] end connection 127.0.0.4:40081 (4 connections now open) m31001| Wed Jan 16 17:23:06.698 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Wed Jan 16 17:23:06.698 [initandlisten] connection accepted from 127.0.0.3:56820 #9 (5 connections now open) m31002| Wed Jan 16 17:23:06.708 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Wed Jan 16 17:23:06.708 [initandlisten] connection accepted from 127.0.0.4:46641 #10 (6 connections now open) m31002| Wed Jan 16 17:23:07.596 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Wed Jan 16 17:23:07.596 [initandlisten] connection accepted from 127.0.0.4:39858 #11 (7 connections now open) m31001| Wed Jan 16 17:23:07.698 [rsSync] replSet SECONDARY m31000| Wed Jan 16 17:23:07.700 [slaveTracking] build index local.slaves { _id: 1 } m31000| Wed Jan 16 17:23:07.700 [slaveTracking] build index done. scanned 0 total records. 0 secs m31002| Wed Jan 16 17:23:08.493 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Wed Jan 16 17:23:08.596 [rsSync] replSet SECONDARY ---- Started set! ---- Caught exception TypeError: Cannot read property '0' of undefined, this is normal after reconfig. Running: sudo bash -c dig +short undefined > /tmp/output Wed Jan 16 17:23:08.643 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Wed Jan 16 17:23:08.676 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dport 31002 0xFFFF flowid 1:11 > /tmp/output Wed Jan 16 17:23:08.690 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dport 31002 0xFFFF flowid 1:11 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Wed Jan 16 17:23:08.704 shell: started program sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 500ms > /tmp/output Wed Jan 16 17:23:08.716 shell: started program sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 500ms > /tmp/output Running: sudo bash -c dig +short undefined > /tmp/output Wed Jan 16 17:23:08.730 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Wed Jan 16 17:23:08.762 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.3 match ip dport 31000 0xFFFF flowid 1:12 > /tmp/output Wed Jan 16 17:23:08.775 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.3 match ip dport 31000 0xFFFF flowid 1:12 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Wed Jan 16 17:23:08.786 shell: started program sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 500ms > /tmp/output Wed Jan 16 17:23:08.797 shell: started program sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 500ms > /tmp/output Running: sudo bash -c dig +short undefined > /tmp/output Wed Jan 16 17:23:08.805 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:13 htb rate 1bps ceil 1000Mbps > /tmp/output Wed Jan 16 17:23:08.834 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:13 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.4 match ip dport 31001 0xFFFF flowid 1:13 > /tmp/output Wed Jan 16 17:23:08.846 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.4 match ip dport 31001 0xFFFF flowid 1:13 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:13 handle 13: netem > /tmp/output Wed Jan 16 17:23:08.859 shell: started program sudo bash -c tc qdisc add dev lo parent 1:13 handle 13: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:13 handle 13: netem delay 500ms > /tmp/output Wed Jan 16 17:23:08.873 shell: started program sudo bash -c tc qdisc change dev lo parent 1:13 handle 13: netem delay 500ms > /tmp/output Running: sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output Wed Jan 16 17:23:08.886 shell: started program sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 1423 ver 3.17 Sent 245626 bytes 1423 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 11: parent 1:11 limit 1000 delay 500.0ms Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 12: parent 1:12 limit 1000 delay 500.0ms Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 13: parent 1:13 limit 1000 delay 500.0ms Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 m31000| Wed Jan 16 17:23:09.239 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Wed Jan 16 17:23:09.739 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Wed Jan 16 17:23:09.858 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY Running: sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output Wed Jan 16 17:23:13.900 shell: started program sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output m31002| Wed Jan 16 17:23:17.240 [conn2] end connection 127.0.0.2:60525 (2 connections now open) m31002| Wed Jan 16 17:23:17.740 [initandlisten] connection accepted from 127.0.0.2:46978 #4 (3 connections now open) m31002| Wed Jan 16 17:23:17.859 [conn3] end connection 127.0.0.3:38174 (2 connections now open) m31002| Wed Jan 16 17:23:17.859 [initandlisten] connection accepted from 127.0.0.3:44836 #5 (3 connections now open) m31000| Wed Jan 16 17:23:18.915 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.ns, filling with zeroes... m31000| Wed Jan 16 17:23:19.068 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.ns, size: 16MB, took 0.152 secs m31000| Wed Jan 16 17:23:19.068 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.0, filling with zeroes... m31000| Wed Jan 16 17:23:19.210 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.0, size: 16MB, took 0.141 secs m31000| Wed Jan 16 17:23:19.211 [conn1] build index foo.bar { _id: 1 } m31000| Wed Jan 16 17:23:19.211 [conn1] build index done. scanned 0 total records. 0 secs m31000| Wed Jan 16 17:23:19.211 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:296345 296ms m31002| Wed Jan 16 17:23:19.212 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.ns, filling with zeroes... m31002| Wed Jan 16 17:23:19.377 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.ns, size: 16MB, took 0.165 secs m31002| Wed Jan 16 17:23:19.378 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.0, filling with zeroes... m31002| Wed Jan 16 17:23:19.520 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.0, size: 16MB, took 0.141 secs m31002| Wed Jan 16 17:23:19.520 [repl writer worker 1] build index foo.bar { _id: 1 } m31002| Wed Jan 16 17:23:19.521 [repl writer worker 1] build index done. scanned 0 total records. 0 secs STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 43, "optime" : { "t" : 1358374999000, "i" : 10000 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 32, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 32, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 30, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:19Z"), "pingMs" : 500 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 42, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358374999000, "i" : 9786 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 29, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:19Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 41, "optime" : { "t" : 1358374999000, "i" : 10000 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:23:20.995 [conn3] end connection 127.0.0.4:34256 (2 connections now open) m31001| Wed Jan 16 17:23:21.495 [initandlisten] connection accepted from 127.0.0.4:56046 #5 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 48, "optime" : { "t" : 1358375004000, "i" : 1317 }, "optimeDate" : ISODate("2013-01-16T22:23:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1358374999000, "i" : 10000 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 35, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:23Z"), "pingMs" : 500 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358375003000, "i" : 7364 }, "optimeDate" : ISODate("2013-01-16T22:23:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:23Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 34, "optime" : { "t" : 1358374999000, "i" : 10000 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:22Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358375004000, "i" : 1317 }, "optimeDate" : ISODate("2013-01-16T22:23:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:23:24.859 [rsHealthPoll] DBClientCursor::init call() failed m31001| Wed Jan 16 17:23:24.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31001| Wed Jan 16 17:23:25.859 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31001| Wed Jan 16 17:23:25.859 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31000| Wed Jan 16 17:23:27.242 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 52, "optime" : { "t" : 1358375008000, "i" : 2814 }, "optimeDate" : ISODate("2013-01-16T22:23:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 41, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 41, "optime" : { "t" : 1358375007000, "i" : 4672 }, "optimeDate" : ISODate("2013-01-16T22:23:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358375007000, "i" : 5896 }, "optimeDate" : ISODate("2013-01-16T22:23:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 38, "optime" : { "t" : 1358375004000, "i" : 1317 }, "optimeDate" : ISODate("2013-01-16T22:23:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:27Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358375008000, "i" : 2673 }, "optimeDate" : ISODate("2013-01-16T22:23:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:29.243 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:23:31.243 [conn4] end connection 127.0.0.2:51015 (2 connections now open) m31001| Wed Jan 16 17:23:31.243 [initandlisten] connection accepted from 127.0.0.2:37139 #6 (3 connections now open) m31000| Wed Jan 16 17:23:31.243 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 56, "optime" : { "t" : 1358375012000, "i" : 4104 }, "optimeDate" : ISODate("2013-01-16T22:23:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358375008000, "i" : 2814 }, "optimeDate" : ISODate("2013-01-16T22:23:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 43, "optime" : { "t" : 1358375011000, "i" : 4634 }, "optimeDate" : ISODate("2013-01-16T22:23:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 42, "optime" : { "t" : 1358375008000, "i" : 2814 }, "optimeDate" : ISODate("2013-01-16T22:23:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 42, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:31Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 54, "optime" : { "t" : 1358375012000, "i" : 4104 }, "optimeDate" : ISODate("2013-01-16T22:23:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:33.244 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:23:34.498 [conn5] end connection 127.0.0.4:53839 (6 connections now open) m31000| Wed Jan 16 17:23:34.498 [initandlisten] connection accepted from 127.0.0.4:58770 #12 (7 connections now open) m31000| Wed Jan 16 17:23:35.244 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 60, "optime" : { "t" : 1358375016000, "i" : 5433 }, "optimeDate" : ISODate("2013-01-16T22:23:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358375012000, "i" : 4104 }, "optimeDate" : ISODate("2013-01-16T22:23:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : { "t" : 1358375015000, "i" : 3012 }, "optimeDate" : ISODate("2013-01-16T22:23:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 46, "optime" : { "t" : 1358375016000, "i" : 4122 }, "optimeDate" : ISODate("2013-01-16T22:23:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 58, "optime" : { "t" : 1358375016000, "i" : 5324 }, "optimeDate" : ISODate("2013-01-16T22:23:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:37.245 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:23:37.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:23:39.245 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 64, "optime" : { "t" : 1358375020000, "i" : 6793 }, "optimeDate" : ISODate("2013-01-16T22:23:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 53, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 53, "optime" : { "t" : 1358375020000, "i" : 1624 }, "optimeDate" : ISODate("2013-01-16T22:23:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 63, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358375019000, "i" : 1891 }, "optimeDate" : ISODate("2013-01-16T22:23:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 50, "optime" : { "t" : 1358375020000, "i" : 4226 }, "optimeDate" : ISODate("2013-01-16T22:23:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:39Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 62, "optime" : { "t" : 1358375020000, "i" : 6793 }, "optimeDate" : ISODate("2013-01-16T22:23:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:41.245 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:23:43.246 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 68, "optime" : { "t" : 1358375024000, "i" : 8062 }, "optimeDate" : ISODate("2013-01-16T22:23:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 57, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 57, "optime" : { "t" : 1358375020000, "i" : 6793 }, "optimeDate" : ISODate("2013-01-16T22:23:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358375023000, "i" : 704 }, "optimeDate" : ISODate("2013-01-16T22:23:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 54, "optime" : { "t" : 1358375024000, "i" : 4110 }, "optimeDate" : ISODate("2013-01-16T22:23:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 54, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358375024000, "i" : 8062 }, "optimeDate" : ISODate("2013-01-16T22:23:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:45.246 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:23:46.679 [rsBackgroundSync] Socket recv() timeout 127.0.0.2:31000 m31001| Wed Jan 16 17:23:46.679 [rsBackgroundSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [3] server [127.0.0.2:31000] m31001| Wed Jan 16 17:23:46.679 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Wed Jan 16 17:23:47.247 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:23:47.864 [conn5] end connection 127.0.0.3:44836 (2 connections now open) m31002| Wed Jan 16 17:23:47.864 [initandlisten] connection accepted from 127.0.0.3:53591 #6 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 73, "optime" : { "t" : 1358375029000, "i" : 1141 }, "optimeDate" : ISODate("2013-01-16T22:23:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 62, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 62, "optime" : { "t" : 1358375024000, "i" : 8062 }, "optimeDate" : ISODate("2013-01-16T22:23:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358375024000, "i" : 8062 }, "optimeDate" : ISODate("2013-01-16T22:23:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 59, "optime" : { "t" : 1358375028000, "i" : 4600 }, "optimeDate" : ISODate("2013-01-16T22:23:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358375029000, "i" : 1141 }, "optimeDate" : ISODate("2013-01-16T22:23:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:23:49.247 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:23:50.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:23:51.248 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:23:53.248 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 77, "optime" : { "t" : 1358375033000, "i" : 2381 }, "optimeDate" : ISODate("2013-01-16T22:23:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358375032000, "i" : 4912 }, "optimeDate" : ISODate("2013-01-16T22:23:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 64, "optime" : { "t" : 1358375029000, "i" : 1141 }, "optimeDate" : ISODate("2013-01-16T22:23:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 63, "optime" : { "t" : 1358375032000, "i" : 3181 }, "optimeDate" : ISODate("2013-01-16T22:23:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 63, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:51Z"), "pingMs" : 500, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 75, "optime" : { "t" : 1358375033000, "i" : 2381 }, "optimeDate" : ISODate("2013-01-16T22:23:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:23:55.246 [conn4] end connection 127.0.0.2:46978 (2 connections now open) m31000| Wed Jan 16 17:23:55.248 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:23:55.746 [initandlisten] connection accepted from 127.0.0.2:55357 #7 (3 connections now open) m31001| Wed Jan 16 17:23:56.679 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Wed Jan 16 17:23:56.679 [initandlisten] connection accepted from 127.0.0.3:40699 #8 (4 connections now open) m31001| Wed Jan 16 17:23:56.680 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.ns, filling with zeroes... m31001| Wed Jan 16 17:23:56.826 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.ns, size: 16MB, took 0.145 secs m31001| Wed Jan 16 17:23:56.826 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.0, filling with zeroes... m31000| Wed Jan 16 17:23:56.918 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.1, filling with zeroes... m31001| Wed Jan 16 17:23:56.977 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.0, size: 16MB, took 0.15 secs m31001| Wed Jan 16 17:23:56.977 [repl writer worker 1] build index foo.bar { _id: 1 } m31001| Wed Jan 16 17:23:56.978 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Wed Jan 16 17:23:57.017 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31002| Wed Jan 16 17:23:57.017 [initandlisten] connection accepted from 127.0.0.3:52687 #9 (5 connections now open) m31000| Wed Jan 16 17:23:57.116 [initandlisten] connection accepted from 127.0.0.4:38720 #13 (8 connections now open) m31002| Wed Jan 16 17:23:57.116 [rsGhostSync] handshake between 1 and 127.0.0.2:31000 m31000| Wed Jan 16 17:23:57.194 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.1, size: 32MB, took 0.275 secs m31000| Wed Jan 16 17:23:57.194 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:276172 276ms m31002| Wed Jan 16 17:23:57.208 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.1, filling with zeroes... m31000| Wed Jan 16 17:23:57.248 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 81, "optime" : { "t" : 1358375037000, "i" : 4844 }, "optimeDate" : ISODate("2013-01-16T22:23:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 70, "optime" : { "t" : 1358374999000, "i" : 6117 }, "optimeDate" : ISODate("2013-01-16T22:23:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 70, "optime" : { "t" : 1358375033000, "i" : 2381 }, "optimeDate" : ISODate("2013-01-16T22:23:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:57Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 80, "optime" : { "t" : 1358375002000, "i" : 3 }, "optimeDate" : ISODate("2013-01-16T22:23:22Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 68, "optime" : { "t" : 1358375033000, "i" : 2381 }, "optimeDate" : ISODate("2013-01-16T22:23:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:23:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 67, "optime" : { "t" : 1358375036000, "i" : 1749 }, "optimeDate" : ISODate("2013-01-16T22:23:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358375036000, "i" : 5156 }, "optimeDate" : ISODate("2013-01-16T22:23:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:23:57.504 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.1, size: 32MB, took 0.295 secs m31002| Wed Jan 16 17:23:58.116 [slaveTracking] build index local.slaves { _id: 1 } m31002| Wed Jan 16 17:23:58.117 [slaveTracking] build index done. scanned 0 total records. 0 secs m31001| Wed Jan 16 17:23:59.000 [conn5] end connection 127.0.0.4:56046 (2 connections now open) m31000| Wed Jan 16 17:23:59.248 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:23:59.501 [initandlisten] connection accepted from 127.0.0.4:43241 #7 (3 connections now open) m31001| Wed Jan 16 17:23:59.901 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.1, filling with zeroes... m31001| Wed Jan 16 17:24:00.172 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.1, size: 32MB, took 0.271 secs m31001| Wed Jan 16 17:24:01.249 [conn6] end connection 127.0.0.2:37139 (2 connections now open) m31001| Wed Jan 16 17:24:01.249 [initandlisten] connection accepted from 127.0.0.2:39351 #8 (3 connections now open) m31000| Wed Jan 16 17:24:01.249 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 85, "optime" : { "t" : 1358375041000, "i" : 4890 }, "optimeDate" : ISODate("2013-01-16T22:24:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 74, "optime" : { "t" : 1358375040000, "i" : 3171 }, "optimeDate" : ISODate("2013-01-16T22:24:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 74, "optime" : { "t" : 1358375040000, "i" : 2802 }, "optimeDate" : ISODate("2013-01-16T22:24:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:01Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 84, "optime" : { "t" : 1358375040000, "i" : 3171 }, "optimeDate" : ISODate("2013-01-16T22:24:00Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358375037000, "i" : 4844 }, "optimeDate" : ISODate("2013-01-16T22:23:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 71, "optime" : { "t" : 1358375040000, "i" : 793 }, "optimeDate" : ISODate("2013-01-16T22:24:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358375032000, "i" : 17 }, "optimeDate" : ISODate("2013-01-16T22:23:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:23:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:23:59Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358375041000, "i" : 4890 }, "optimeDate" : ISODate("2013-01-16T22:24:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:24:01.867 [rsMgr] not electing self, we are not freshest m31000| Wed Jan 16 17:24:03.249 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:03.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:24:04.503 [conn12] end connection 127.0.0.4:58770 (7 connections now open) m31000| Wed Jan 16 17:24:04.503 [initandlisten] connection accepted from 127.0.0.4:42231 #14 (8 connections now open) m31000| Wed Jan 16 17:24:04.859 [conn4] end connection 127.0.0.3:43374 (7 connections now open) m31000| Wed Jan 16 17:24:05.249 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 89, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358375044000, "i" : 1667 }, "optimeDate" : ISODate("2013-01-16T22:24:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358375045000, "i" : 5708 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:05Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 88, "optime" : { "t" : 1358375044000, "i" : 1667 }, "optimeDate" : ISODate("2013-01-16T22:24:04Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358375041000, "i" : 4890 }, "optimeDate" : ISODate("2013-01-16T22:24:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 75, "optime" : { "t" : 1358375041000, "i" : 4890 }, "optimeDate" : ISODate("2013-01-16T22:24:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 75, "optime" : { "t" : 1358375041000, "i" : 4890 }, "optimeDate" : ISODate("2013-01-16T22:24:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 87, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:07.250 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:07.869 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:24:09.250 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 94, "optime" : { "t" : 1358375050000, "i" : 606 }, "optimeDate" : ISODate("2013-01-16T22:24:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:10Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 93, "optime" : { "t" : 1358375049000, "i" : 3661 }, "optimeDate" : ISODate("2013-01-16T22:24:09Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358375049000, "i" : 6877 }, "optimeDate" : ISODate("2013-01-16T22:24:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 80, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 80, "optime" : { "t" : 1358375045000, "i" : 6584 }, "optimeDate" : ISODate("2013-01-16T22:24:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:09Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 92, "optime" : { "t" : 1358375050000, "i" : 606 }, "optimeDate" : ISODate("2013-01-16T22:24:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:11.251 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:13.251 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:13.869 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 98, "optime" : { "t" : 1358375054000, "i" : 2080 }, "optimeDate" : ISODate("2013-01-16T22:24:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 87, "optime" : { "t" : 1358375050000, "i" : 606 }, "optimeDate" : ISODate("2013-01-16T22:24:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 87, "optime" : { "t" : 1358375053000, "i" : 1335 }, "optimeDate" : ISODate("2013-01-16T22:24:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:14Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 97, "optime" : { "t" : 1358375053000, "i" : 6353 }, "optimeDate" : ISODate("2013-01-16T22:24:13Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 85, "optime" : { "t" : 1358375053000, "i" : 6353 }, "optimeDate" : ISODate("2013-01-16T22:24:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 84, "optime" : { "t" : 1358375050000, "i" : 606 }, "optimeDate" : ISODate("2013-01-16T22:24:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 84, "optime" : { "t" : 1358375050000, "i" : 606 }, "optimeDate" : ISODate("2013-01-16T22:24:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:13Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 96, "optime" : { "t" : 1358375054000, "i" : 2080 }, "optimeDate" : ISODate("2013-01-16T22:24:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:15.251 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:16.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:24:17.252 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:24:17.870 [conn6] end connection 127.0.0.3:53591 (4 connections now open) m31002| Wed Jan 16 17:24:17.870 [initandlisten] connection accepted from 127.0.0.3:47356 #10 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 102, "optime" : { "t" : 1358375058000, "i" : 3438 }, "optimeDate" : ISODate("2013-01-16T22:24:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 91, "optime" : { "t" : 1358375054000, "i" : 2080 }, "optimeDate" : ISODate("2013-01-16T22:24:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 91, "optime" : { "t" : 1358375058000, "i" : 2365 }, "optimeDate" : ISODate("2013-01-16T22:24:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:18Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 101, "optime" : { "t" : 1358375057000, "i" : 5906 }, "optimeDate" : ISODate("2013-01-16T22:24:17Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 89, "optime" : { "t" : 1358375057000, "i" : 5256 }, "optimeDate" : ISODate("2013-01-16T22:24:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 88, "optime" : { "t" : 1358375054000, "i" : 2080 }, "optimeDate" : ISODate("2013-01-16T22:24:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 88, "optime" : { "t" : 1358375054000, "i" : 2080 }, "optimeDate" : ISODate("2013-01-16T22:24:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:17Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 100, "optime" : { "t" : 1358375058000, "i" : 3438 }, "optimeDate" : ISODate("2013-01-16T22:24:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:19.252 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:19.871 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:24:21.252 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 106, "optime" : { "t" : 1358375062000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T22:24:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 95, "optime" : { "t" : 1358375058000, "i" : 3438 }, "optimeDate" : ISODate("2013-01-16T22:24:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 95, "optime" : { "t" : 1358375058000, "i" : 3438 }, "optimeDate" : ISODate("2013-01-16T22:24:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:22Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 105, "optime" : { "t" : 1358375061000, "i" : 4593 }, "optimeDate" : ISODate("2013-01-16T22:24:21Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 93, "optime" : { "t" : 1358375061000, "i" : 4407 }, "optimeDate" : ISODate("2013-01-16T22:24:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 92, "optime" : { "t" : 1358375062000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T22:24:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 92, "optime" : { "t" : 1358375061000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:24:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 104, "optime" : { "t" : 1358375062000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T22:24:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:23.253 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:25.253 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:25.871 [rsMgr] not electing self, we are not freshest m31000| Wed Jan 16 17:24:25.980 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:459742 229ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 110, "optime" : { "t" : 1358375066000, "i" : 7156 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 99, "optime" : { "t" : 1358375062000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T22:24:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 99, "optime" : { "t" : 1358375065000, "i" : 2114 }, "optimeDate" : ISODate("2013-01-16T22:24:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:26Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 109, "optime" : { "t" : 1358375065000, "i" : 2114 }, "optimeDate" : ISODate("2013-01-16T22:24:25Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 97, "optime" : { "t" : 1358375065000, "i" : 2114 }, "optimeDate" : ISODate("2013-01-16T22:24:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 96, "optime" : { "t" : 1358375066000, "i" : 6548 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 96, "optime" : { "t" : 1358375062000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T22:24:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:25Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 108, "optime" : { "t" : 1358375066000, "i" : 6443 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:27.254 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:29.254 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:29.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 114, "optime" : { "t" : 1358375070000, "i" : 5976 }, "optimeDate" : ISODate("2013-01-16T22:24:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 103, "optime" : { "t" : 1358375066000, "i" : 7156 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 103, "optime" : { "t" : 1358375066000, "i" : 7156 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:30Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 113, "optime" : { "t" : 1358375069000, "i" : 3907 }, "optimeDate" : ISODate("2013-01-16T22:24:29Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 101, "optime" : { "t" : 1358375069000, "i" : 2933 }, "optimeDate" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 100, "optime" : { "t" : 1358375070000, "i" : 4218 }, "optimeDate" : ISODate("2013-01-16T22:24:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 100, "optime" : { "t" : 1358375066000, "i" : 7156 }, "optimeDate" : ISODate("2013-01-16T22:24:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:29Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 112, "optime" : { "t" : 1358375070000, "i" : 5976 }, "optimeDate" : ISODate("2013-01-16T22:24:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:24:30.859 [rsMgr] not electing self, we are not freshest m31001| Wed Jan 16 17:24:31.254 [conn8] end connection 127.0.0.2:39351 (2 connections now open) m31001| Wed Jan 16 17:24:31.255 [initandlisten] connection accepted from 127.0.0.2:45238 #9 (3 connections now open) m31000| Wed Jan 16 17:24:31.255 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:32.316 [slaveTracking] update local.slaves query: { _id: ObjectId('50f72849a8d5162066e60c36'), host: "127.0.0.3", ns: "local.oplog.rs" } update: { $set: { syncedTo: Timestamp 1358375066000|7156 } } nscanned:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:196 1333ms m31002| Wed Jan 16 17:24:33.251 [conn7] end connection 127.0.0.2:55357 (4 connections now open) m31000| Wed Jan 16 17:24:33.255 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:24:33.751 [initandlisten] connection accepted from 127.0.0.2:34548 #11 (5 connections now open) m31000| Wed Jan 16 17:24:34.508 [conn14] end connection 127.0.0.4:42231 (6 connections now open) m31000| Wed Jan 16 17:24:34.508 [initandlisten] connection accepted from 127.0.0.4:39710 #15 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 118, "optime" : { "t" : 1358375074000, "i" : 7177 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 107, "optime" : { "t" : 1358375070000, "i" : 5976 }, "optimeDate" : ISODate("2013-01-16T22:24:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 107, "optime" : { "t" : 1358375073000, "i" : 202 }, "optimeDate" : ISODate("2013-01-16T22:24:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:34Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358375074000, "i" : 5 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 105, "optime" : { "t" : 1358375073000, "i" : 1475 }, "optimeDate" : ISODate("2013-01-16T22:24:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 104, "optime" : { "t" : 1358375074000, "i" : 4610 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 104, "optime" : { "t" : 1358375074000, "i" : 5 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358375074000, "i" : 7177 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:35.256 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:35.873 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31001| Wed Jan 16 17:24:37.006 [conn7] end connection 127.0.0.4:43241 (2 connections now open) m31000| Wed Jan 16 17:24:37.017 [conn9] end connection 127.0.0.3:56820 (6 connections now open) m31000| Wed Jan 16 17:24:37.256 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:37.506 [initandlisten] connection accepted from 127.0.0.4:51642 #10 (3 connections now open) m31000| Wed Jan 16 17:24:38.242 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:261311 130ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 122, "optime" : { "t" : 1358375078000, "i" : 8624 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358375074000, "i" : 7177 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358375078000, "i" : 6995 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:38Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 121, "optime" : { "t" : 1358375078000, "i" : 423 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 109, "optime" : { "t" : 1358375077000, "i" : 502 }, "optimeDate" : ISODate("2013-01-16T22:24:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 108, "optime" : { "t" : 1358375078000, "i" : 4837 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 108, "optime" : { "t" : 1358375074000, "i" : 7177 }, "optimeDate" : ISODate("2013-01-16T22:24:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:37Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 120, "optime" : { "t" : 1358375078000, "i" : 8624 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:39.256 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:41.257 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:41.874 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31001| Wed Jan 16 17:24:42.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 127, "optime" : { "t" : 1358375083000, "i" : 1122 }, "optimeDate" : ISODate("2013-01-16T22:24:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358375078000, "i" : 8624 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358375078000, "i" : 8624 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:43Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 126, "optime" : { "t" : 1358375082000, "i" : 250 }, "optimeDate" : ISODate("2013-01-16T22:24:42Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 114, "optime" : { "t" : 1358375078000, "i" : 8624 }, "optimeDate" : ISODate("2013-01-16T22:24:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 113, "optime" : { "t" : 1358375082000, "i" : 4284 }, "optimeDate" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 113, "optime" : { "t" : 1358375082000, "i" : 250 }, "optimeDate" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 125, "optime" : { "t" : 1358375083000, "i" : 1122 }, "optimeDate" : ISODate("2013-01-16T22:24:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:43.257 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:45.257 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:47.257 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 131, "optime" : { "t" : 1358375087000, "i" : 2750 }, "optimeDate" : ISODate("2013-01-16T22:24:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 120, "optime" : { "t" : 1358375086000, "i" : 2 }, "optimeDate" : ISODate("2013-01-16T22:24:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 120, "optime" : { "t" : 1358375086000, "i" : 735 }, "optimeDate" : ISODate("2013-01-16T22:24:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:47Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 130, "optime" : { "t" : 1358375086000, "i" : 2 }, "optimeDate" : ISODate("2013-01-16T22:24:46Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 118, "optime" : { "t" : 1358375083000, "i" : 1122 }, "optimeDate" : ISODate("2013-01-16T22:24:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 117, "optime" : { "t" : 1358375086000, "i" : 3161 }, "optimeDate" : ISODate("2013-01-16T22:24:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358375083000, "i" : 1122 }, "optimeDate" : ISODate("2013-01-16T22:24:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:45Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 129, "optime" : { "t" : 1358375087000, "i" : 2750 }, "optimeDate" : ISODate("2013-01-16T22:24:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:24:47.875 [conn10] end connection 127.0.0.3:47356 (4 connections now open) m31002| Wed Jan 16 17:24:47.875 [initandlisten] connection accepted from 127.0.0.3:51069 #12 (5 connections now open) m31001| Wed Jan 16 17:24:47.875 [rsMgr] not electing self, we are not freshest m31000| Wed Jan 16 17:24:49.258 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:51.258 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 135, "optime" : { "t" : 1358375091000, "i" : 3650 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 124, "optime" : { "t" : 1358375090000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-16T22:24:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 124, "optime" : { "t" : 1358375091000, "i" : 1942 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:51Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 134, "optime" : { "t" : 1358375091000, "i" : 1034 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 122, "optime" : { "t" : 1358375087000, "i" : 2750 }, "optimeDate" : ISODate("2013-01-16T22:24:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 121, "optime" : { "t" : 1358375090000, "i" : 2279 }, "optimeDate" : ISODate("2013-01-16T22:24:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 121, "optime" : { "t" : 1358375087000, "i" : 2750 }, "optimeDate" : ISODate("2013-01-16T22:24:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 133, "optime" : { "t" : 1358375091000, "i" : 3066 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:24:53.258 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:24:53.877 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:24:55.259 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 139, "optime" : { "t" : 1358375095000, "i" : 5009 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 128, "optime" : { "t" : 1358375094000, "i" : 2 }, "optimeDate" : ISODate("2013-01-16T22:24:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 128, "optime" : { "t" : 1358375091000, "i" : 3650 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:55Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 138, "optime" : { "t" : 1358375095000, "i" : 1041 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 126, "optime" : { "t" : 1358375091000, "i" : 3650 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 125, "optime" : { "t" : 1358375094000, "i" : 938 }, "optimeDate" : ISODate("2013-01-16T22:24:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 125, "optime" : { "t" : 1358375091000, "i" : 3650 }, "optimeDate" : ISODate("2013-01-16T22:24:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 137, "optime" : { "t" : 1358375095000, "i" : 4869 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:24:55.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:24:57.259 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:24:59.259 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 143, "optime" : { "t" : 1358375099000, "i" : 5619 }, "optimeDate" : ISODate("2013-01-16T22:24:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 132, "optime" : { "t" : 1358375098000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:24:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 132, "optime" : { "t" : 1358375098000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:24:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:59Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358375099000, "i" : 1145 }, "optimeDate" : ISODate("2013-01-16T22:24:59Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 130, "optime" : { "t" : 1358375095000, "i" : 5009 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:24:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 129, "optime" : { "t" : 1358375095000, "i" : 5009 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 129, "optime" : { "t" : 1358375095000, "i" : 5009 }, "optimeDate" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:24:57Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 141, "optime" : { "t" : 1358375099000, "i" : 5619 }, "optimeDate" : ISODate("2013-01-16T22:24:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:24:59.878 [rsMgr] not electing self, we are not freshest m31001| Wed Jan 16 17:25:01.259 [conn9] end connection 127.0.0.2:45238 (2 connections now open) m31001| Wed Jan 16 17:25:01.260 [initandlisten] connection accepted from 127.0.0.2:41902 #11 (3 connections now open) m31000| Wed Jan 16 17:25:01.260 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:03.260 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 147, "optime" : { "t" : 1358375103000, "i" : 7067 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 136, "optime" : { "t" : 1358375102000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:25:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 136, "optime" : { "t" : 1358375103000, "i" : 6343 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:03Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358375103000, "i" : 1535 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 134, "optime" : { "t" : 1358375103000, "i" : 6920 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 133, "optime" : { "t" : 1358375099000, "i" : 5619 }, "optimeDate" : ISODate("2013-01-16T22:24:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 133, "optime" : { "t" : 1358375099000, "i" : 5619 }, "optimeDate" : ISODate("2013-01-16T22:24:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:03Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 145, "optime" : { "t" : 1358375103000, "i" : 6920 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:04.513 [conn15] end connection 127.0.0.4:39710 (5 connections now open) m31000| Wed Jan 16 17:25:04.513 [initandlisten] connection accepted from 127.0.0.4:50443 #16 (6 connections now open) m31000| Wed Jan 16 17:25:05.261 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:05.879 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:25:06.719 [conn7] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:38016] m31000| Wed Jan 16 17:25:07.261 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 152, "optime" : { "t" : 1358375108000, "i" : 645 }, "optimeDate" : ISODate("2013-01-16T22:25:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 141, "optime" : { "t" : 1358375106000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:25:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 141, "optime" : { "t" : 1358375103000, "i" : 7067 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:08Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:24:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 151, "optime" : { "t" : 1358375107000, "i" : 2625 }, "optimeDate" : ISODate("2013-01-16T22:25:07Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 139, "optime" : { "t" : 1358375107000, "i" : 8498 }, "optimeDate" : ISODate("2013-01-16T22:25:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 138, "optime" : { "t" : 1358375103000, "i" : 7067 }, "optimeDate" : ISODate("2013-01-16T22:25:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 138, "optime" : { "t" : 1358375107000, "i" : 2625 }, "optimeDate" : ISODate("2013-01-16T22:25:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:07Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 150, "optime" : { "t" : 1358375107000, "i" : 8942 }, "optimeDate" : ISODate("2013-01-16T22:25:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:25:08.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Wed Jan 16 17:25:09.261 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:25:11.256 [conn11] end connection 127.0.0.2:34548 (4 connections now open) m31000| Wed Jan 16 17:25:11.261 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:25:11.756 [initandlisten] connection accepted from 127.0.0.2:54351 #13 (5 connections now open) m31001| Wed Jan 16 17:25:11.879 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 156, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 145, "optime" : { "t" : 1358375111000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 145, "optime" : { "t" : 1358375111000, "i" : 5753 }, "optimeDate" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:12Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 155, "optime" : { "t" : 1358375111000, "i" : 1645 }, "optimeDate" : ISODate("2013-01-16T22:25:11Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 143, "optime" : { "t" : 1358375111000, "i" : 7073 }, "optimeDate" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 142, "optime" : { "t" : 1358375108000, "i" : 645 }, "optimeDate" : ISODate("2013-01-16T22:25:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358375108000, "i" : 645 }, "optimeDate" : ISODate("2013-01-16T22:25:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:11Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:13.262 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:15.011 [conn10] end connection 127.0.0.4:51642 (2 connections now open) m31000| Wed Jan 16 17:25:15.262 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:15.511 [initandlisten] connection accepted from 127.0.0.4:57053 #12 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 160, "optime" : { "t" : 1358375116000, "i" : 3771 }, "optimeDate" : ISODate("2013-01-16T22:25:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 149, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 149, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:16Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358375115000, "i" : 2 }, "optimeDate" : ISODate("2013-01-16T22:25:15Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358375115000, "i" : 4974 }, "optimeDate" : ISODate("2013-01-16T22:25:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 146, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358375112000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:25:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:15Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358375116000, "i" : 3240 }, "optimeDate" : ISODate("2013-01-16T22:25:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:17.263 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:25:17.880 [conn12] end connection 127.0.0.3:51069 (4 connections now open) m31002| Wed Jan 16 17:25:17.880 [initandlisten] connection accepted from 127.0.0.3:59032 #14 (5 connections now open) m31001| Wed Jan 16 17:25:17.881 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:25:19.263 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 164, "optime" : { "t" : 1358375120000, "i" : 4885 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358375116000, "i" : 3771 }, "optimeDate" : ISODate("2013-01-16T22:25:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358375116000, "i" : 3771 }, "optimeDate" : ISODate("2013-01-16T22:25:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:20Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358375120000, "i" : 2736 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 151, "optime" : { "t" : 1358375119000, "i" : 3777 }, "optimeDate" : ISODate("2013-01-16T22:25:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 150, "optime" : { "t" : 1358375120000, "i" : 4628 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 150, "optime" : { "t" : 1358375119000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-16T22:25:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358375120000, "i" : 4671 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:21.264 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:21.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31001| Wed Jan 16 17:25:22.859 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:25:23.264 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 168, "optime" : { "t" : 1358375124000, "i" : 6296 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358375120000, "i" : 4885 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358375124000, "i" : 2187 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:24Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358375124000, "i" : 2572 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 155, "optime" : { "t" : 1358375123000, "i" : 2499 }, "optimeDate" : ISODate("2013-01-16T22:25:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 154, "optime" : { "t" : 1358375124000, "i" : 4019 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358375120000, "i" : 4885 }, "optimeDate" : ISODate("2013-01-16T22:25:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:23Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 166, "optime" : { "t" : 1358375124000, "i" : 5207 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:25.264 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:27.265 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:27.882 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 172, "optime" : { "t" : 1358375128000, "i" : 7502 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 161, "optime" : { "t" : 1358375124000, "i" : 6296 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 161, "optime" : { "t" : 1358375124000, "i" : 6296 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:28Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 171, "optime" : { "t" : 1358375128000, "i" : 3254 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358375127000, "i" : 1404 }, "optimeDate" : ISODate("2013-01-16T22:25:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 158, "optime" : { "t" : 1358375128000, "i" : 4719 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358375124000, "i" : 6296 }, "optimeDate" : ISODate("2013-01-16T22:25:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 170, "optime" : { "t" : 1358375128000, "i" : 7502 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:29.265 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:31.266 [conn11] end connection 127.0.0.2:41902 (2 connections now open) m31001| Wed Jan 16 17:25:31.266 [initandlisten] connection accepted from 127.0.0.2:60717 #13 (3 connections now open) m31000| Wed Jan 16 17:25:31.266 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 176, "optime" : { "t" : 1358375132000, "i" : 8840 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 165, "optime" : { "t" : 1358375128000, "i" : 7502 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 165, "optime" : { "t" : 1358375128000, "i" : 7502 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:32Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358375132000, "i" : 3327 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358375131000, "i" : 205 }, "optimeDate" : ISODate("2013-01-16T22:25:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 162, "optime" : { "t" : 1358375132000, "i" : 4669 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358375128000, "i" : 7502 }, "optimeDate" : ISODate("2013-01-16T22:25:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:31Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358375132000, "i" : 8476 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:33.266 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:33.883 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:25:34.518 [conn16] end connection 127.0.0.4:50443 (4 connections now open) m31000| Wed Jan 16 17:25:34.518 [initandlisten] connection accepted from 127.0.0.4:57893 #17 (5 connections now open) m31001| Wed Jan 16 17:25:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:25:35.267 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:36.445 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:223045 111ms m31000| Wed Jan 16 17:25:36.975 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:210223 105ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 181, "optime" : { "t" : 1358375137000, "i" : 1711 }, "optimeDate" : ISODate("2013-01-16T22:25:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 170, "optime" : { "t" : 1358375132000, "i" : 8840 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 170, "optime" : { "t" : 1358375136000, "i" : 6216 }, "optimeDate" : ISODate("2013-01-16T22:25:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:37Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358375136000, "i" : 2867 }, "optimeDate" : ISODate("2013-01-16T22:25:36Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 168, "optime" : { "t" : 1358375132000, "i" : 8840 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 167, "optime" : { "t" : 1358375136000, "i" : 4651 }, "optimeDate" : ISODate("2013-01-16T22:25:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358375132000, "i" : 8840 }, "optimeDate" : ISODate("2013-01-16T22:25:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:35Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358375137000, "i" : 1711 }, "optimeDate" : ISODate("2013-01-16T22:25:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:37.267 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:39.267 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:39.885 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 185, "optime" : { "t" : 1358375141000, "i" : 2455 }, "optimeDate" : ISODate("2013-01-16T22:25:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358375137000, "i" : 1711 }, "optimeDate" : ISODate("2013-01-16T22:25:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358375137000, "i" : 1711 }, "optimeDate" : ISODate("2013-01-16T22:25:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:41Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358375140000, "i" : 2631 }, "optimeDate" : ISODate("2013-01-16T22:25:40Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 172, "optime" : { "t" : 1358375137000, "i" : 1711 }, "optimeDate" : ISODate("2013-01-16T22:25:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 171, "optime" : { "t" : 1358375140000, "i" : 3030 }, "optimeDate" : ISODate("2013-01-16T22:25:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 171, "optime" : { "t" : 1358375140000, "i" : 2631 }, "optimeDate" : ISODate("2013-01-16T22:25:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 183, "optime" : { "t" : 1358375141000, "i" : 2455 }, "optimeDate" : ISODate("2013-01-16T22:25:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:41.268 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:43.268 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:44.858 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:25:45.268 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 189, "optime" : { "t" : 1358375145000, "i" : 3706 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 178, "optime" : { "t" : 1358375144000, "i" : 1490 }, "optimeDate" : ISODate("2013-01-16T22:25:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 178, "optime" : { "t" : 1358375144000, "i" : 3 }, "optimeDate" : ISODate("2013-01-16T22:25:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:45Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 188, "optime" : { "t" : 1358375144000, "i" : 1490 }, "optimeDate" : ISODate("2013-01-16T22:25:44Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 176, "optime" : { "t" : 1358375141000, "i" : 2455 }, "optimeDate" : ISODate("2013-01-16T22:25:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 175, "optime" : { "t" : 1358375144000, "i" : 2225 }, "optimeDate" : ISODate("2013-01-16T22:25:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358375141000, "i" : 2455 }, "optimeDate" : ISODate("2013-01-16T22:25:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:43Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 187, "optime" : { "t" : 1358375145000, "i" : 3649 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:25:45.886 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:25:47.269 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:25:47.885 [conn14] end connection 127.0.0.3:59032 (4 connections now open) m31002| Wed Jan 16 17:25:47.886 [initandlisten] connection accepted from 127.0.0.3:47939 #15 (5 connections now open) m31002| Wed Jan 16 17:25:49.261 [conn13] end connection 127.0.0.2:54351 (4 connections now open) m31000| Wed Jan 16 17:25:49.269 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 193, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 182, "optime" : { "t" : 1358375145000, "i" : 3706 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 182, "optime" : { "t" : 1358375145000, "i" : 3706 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:49Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 192, "optime" : { "t" : 1358375148000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:25:48Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358375145000, "i" : 3706 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 179, "optime" : { "t" : 1358375148000, "i" : 469 }, "optimeDate" : ISODate("2013-01-16T22:25:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358375145000, "i" : 3706 }, "optimeDate" : ISODate("2013-01-16T22:25:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:25:49.761 [initandlisten] connection accepted from 127.0.0.2:43264 #16 (5 connections now open) m31000| Wed Jan 16 17:25:51.269 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:51.887 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31001| Wed Jan 16 17:25:53.016 [conn12] end connection 127.0.0.4:57053 (2 connections now open) m31000| Wed Jan 16 17:25:53.270 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:25:53.516 [initandlisten] connection accepted from 127.0.0.4:51867 #14 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 197, "optime" : { "t" : 1358375153000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 186, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 186, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:53Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358375153000, "i" : 4119 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 183, "optime" : { "t" : 1358375149000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-16T22:25:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 183, "optime" : { "t" : 1358375152000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:25:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358375153000, "i" : 5504 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:25:55.270 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:57.270 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:25:57.355 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.2, filling with zeroes... STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:25:57.885 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.2, size: 64MB, took 0.529 secs m31000| Wed Jan 16 17:25:57.885 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:530584 530ms m31002| Wed Jan 16 17:25:57.886 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.2, filling with zeroes... m31001| Wed Jan 16 17:25:57.888 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 202, "optime" : { "t" : 1358375158000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:25:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358375153000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358375157000, "i" : 2506 }, "optimeDate" : ISODate("2013-01-16T22:25:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:58Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358375157000, "i" : 3390 }, "optimeDate" : ISODate("2013-01-16T22:25:57Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 189, "optime" : { "t" : 1358375157000, "i" : 3390 }, "optimeDate" : ISODate("2013-01-16T22:25:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:25:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 188, "optime" : { "t" : 1358375153000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 188, "optime" : { "t" : 1358375153000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:25:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:25:57Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358375157000, "i" : 3390 }, "optimeDate" : ISODate("2013-01-16T22:25:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:25:58.420 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.2, size: 64MB, took 0.533 secs m31001| Wed Jan 16 17:25:58.542 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.2, filling with zeroes... m31001| Wed Jan 16 17:25:59.078 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.2, size: 64MB, took 0.536 secs m31000| Wed Jan 16 17:25:59.271 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:01.271 [conn13] end connection 127.0.0.2:60717 (2 connections now open) m31001| Wed Jan 16 17:26:01.271 [initandlisten] connection accepted from 127.0.0.2:54704 #15 (3 connections now open) m31000| Wed Jan 16 17:26:01.271 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:01.739 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:395911 197ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 206, "optime" : { "t" : 1358375162000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:26:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358375158000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:25:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358375158000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:25:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:02Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 205, "optime" : { "t" : 1358375161000, "i" : 3442 }, "optimeDate" : ISODate("2013-01-16T22:26:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 193, "optime" : { "t" : 1358375161000, "i" : 5595 }, "optimeDate" : ISODate("2013-01-16T22:26:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 192, "optime" : { "t" : 1358375158000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:25:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 192, "optime" : { "t" : 1358375158000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:25:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:01Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 204, "optime" : { "t" : 1358375162000, "i" : 2031 }, "optimeDate" : ISODate("2013-01-16T22:26:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:03.272 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:03.889 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:26:04.523 [conn17] end connection 127.0.0.4:57893 (4 connections now open) m31000| Wed Jan 16 17:26:04.523 [initandlisten] connection accepted from 127.0.0.4:41186 #18 (5 connections now open) m31000| Wed Jan 16 17:26:05.272 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 210, "optime" : { "t" : 1358375166000, "i" : 3965 }, "optimeDate" : ISODate("2013-01-16T22:26:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 199, "optime" : { "t" : 1358375162000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:26:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 199, "optime" : { "t" : 1358375162000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:26:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:06Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 209, "optime" : { "t" : 1358375165000, "i" : 2448 }, "optimeDate" : ISODate("2013-01-16T22:26:05Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 197, "optime" : { "t" : 1358375165000, "i" : 4948 }, "optimeDate" : ISODate("2013-01-16T22:26:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 196, "optime" : { "t" : 1358375166000, "i" : 3965 }, "optimeDate" : ISODate("2013-01-16T22:26:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358375165000, "i" : 2448 }, "optimeDate" : ISODate("2013-01-16T22:26:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 208, "optime" : { "t" : 1358375166000, "i" : 3754 }, "optimeDate" : ISODate("2013-01-16T22:26:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:26:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:26:07.272 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:09.273 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:09.890 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 214, "optime" : { "t" : 1358375170000, "i" : 6140 }, "optimeDate" : ISODate("2013-01-16T22:26:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 203, "optime" : { "t" : 1358375166000, "i" : 3965 }, "optimeDate" : ISODate("2013-01-16T22:26:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 203, "optime" : { "t" : 1358375169000, "i" : 1684 }, "optimeDate" : ISODate("2013-01-16T22:26:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:10Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358375169000, "i" : 499 }, "optimeDate" : ISODate("2013-01-16T22:26:09Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358375169000, "i" : 2841 }, "optimeDate" : ISODate("2013-01-16T22:26:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 200, "optime" : { "t" : 1358375170000, "i" : 4611 }, "optimeDate" : ISODate("2013-01-16T22:26:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358375166000, "i" : 3965 }, "optimeDate" : ISODate("2013-01-16T22:26:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:09Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358375170000, "i" : 6129 }, "optimeDate" : ISODate("2013-01-16T22:26:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:11.273 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:13.274 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 218, "optime" : { "t" : 1358375174000, "i" : 7614 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358375170000, "i" : 6140 }, "optimeDate" : ISODate("2013-01-16T22:26:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358375174000, "i" : 6054 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:14Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358375174000, "i" : 3240 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 205, "optime" : { "t" : 1358375173000, "i" : 1641 }, "optimeDate" : ISODate("2013-01-16T22:26:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 204, "optime" : { "t" : 1358375174000, "i" : 4532 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 204, "optime" : { "t" : 1358375170000, "i" : 6140 }, "optimeDate" : ISODate("2013-01-16T22:26:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:13Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358375174000, "i" : 7278 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:15.274 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:15.891 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31001| Wed Jan 16 17:26:16.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:26:17.274 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:26:17.891 [conn15] end connection 127.0.0.3:47939 (4 connections now open) m31002| Wed Jan 16 17:26:17.891 [initandlisten] connection accepted from 127.0.0.3:37325 #17 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 223, "optime" : { "t" : 1358375179000, "i" : 535 }, "optimeDate" : ISODate("2013-01-16T22:26:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358375174000, "i" : 7614 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358375174000, "i" : 7614 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:19Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358375178000, "i" : 5665 }, "optimeDate" : ISODate("2013-01-16T22:26:18Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 210, "optime" : { "t" : 1358375174000, "i" : 7614 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 209, "optime" : { "t" : 1358375178000, "i" : 4243 }, "optimeDate" : ISODate("2013-01-16T22:26:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 209, "optime" : { "t" : 1358375174000, "i" : 7614 }, "optimeDate" : ISODate("2013-01-16T22:26:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358375179000, "i" : 535 }, "optimeDate" : ISODate("2013-01-16T22:26:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:19.275 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:21.275 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:21.893 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 227, "optime" : { "t" : 1358375183000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:26:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358375179000, "i" : 535 }, "optimeDate" : ISODate("2013-01-16T22:26:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358375182000, "i" : 1761 }, "optimeDate" : ISODate("2013-01-16T22:26:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:23Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:25:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358375182000, "i" : 5771 }, "optimeDate" : ISODate("2013-01-16T22:26:22Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 214, "optime" : { "t" : 1358375179000, "i" : 535 }, "optimeDate" : ISODate("2013-01-16T22:26:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 213, "optime" : { "t" : 1358375182000, "i" : 4205 }, "optimeDate" : ISODate("2013-01-16T22:26:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358375179000, "i" : 535 }, "optimeDate" : ISODate("2013-01-16T22:26:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:21Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 225, "optime" : { "t" : 1358375183000, "i" : 634 }, "optimeDate" : ISODate("2013-01-16T22:26:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:23.275 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:25.276 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:26.859 [rsMgr] not electing self, we are not freshest m31002| Wed Jan 16 17:26:27.266 [conn16] end connection 127.0.0.2:43264 (4 connections now open) m31000| Wed Jan 16 17:26:27.276 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 231, "optime" : { "t" : 1358375187000, "i" : 3180 }, "optimeDate" : ISODate("2013-01-16T22:26:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 220, "optime" : { "t" : 1358375186000, "i" : 4784 }, "optimeDate" : ISODate("2013-01-16T22:26:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 220, "optime" : { "t" : 1358375183000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:26:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:27Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 230, "optime" : { "t" : 1358375186000, "i" : 4784 }, "optimeDate" : ISODate("2013-01-16T22:26:26Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358375183000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:26:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 217, "optime" : { "t" : 1358375186000, "i" : 3037 }, "optimeDate" : ISODate("2013-01-16T22:26:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358375183000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:26:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 229, "optime" : { "t" : 1358375187000, "i" : 3180 }, "optimeDate" : ISODate("2013-01-16T22:26:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:26:27.766 [initandlisten] connection accepted from 127.0.0.2:59087 #18 (5 connections now open) m31000| Wed Jan 16 17:26:29.276 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:31.021 [conn14] end connection 127.0.0.4:51867 (2 connections now open) m31001| Wed Jan 16 17:26:31.277 [conn15] end connection 127.0.0.2:54704 (1 connection now open) m31001| Wed Jan 16 17:26:31.277 [initandlisten] connection accepted from 127.0.0.2:48570 #16 (2 connections now open) m31000| Wed Jan 16 17:26:31.277 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 235, "optime" : { "t" : 1358375191000, "i" : 4064 }, "optimeDate" : ISODate("2013-01-16T22:26:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 224, "optime" : { "t" : 1358375190000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:26:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 224, "optime" : { "t" : 1358375187000, "i" : 3180 }, "optimeDate" : ISODate("2013-01-16T22:26:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:31Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358375190000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:26:30Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358375187000, "i" : 3180 }, "optimeDate" : ISODate("2013-01-16T22:26:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 221, "optime" : { "t" : 1358375190000, "i" : 1822 }, "optimeDate" : ISODate("2013-01-16T22:26:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358375187000, "i" : 3180 }, "optimeDate" : ISODate("2013-01-16T22:26:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:29Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358375191000, "i" : 4064 }, "optimeDate" : ISODate("2013-01-16T22:26:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:26:31.521 [initandlisten] connection accepted from 127.0.0.4:33931 #17 (3 connections now open) m31001| Wed Jan 16 17:26:31.895 [rsMgr] not electing self, we are not freshest m31000| Wed Jan 16 17:26:33.277 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:33.436 [slaveTracking] update local.slaves query: { _id: ObjectId('50f72849a8d5162066e60c36'), host: "127.0.0.3", ns: "local.oplog.rs" } update: { $set: { syncedTo: Timestamp 1358375187000|3180 } } nscanned:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:189 1400ms m31000| Wed Jan 16 17:26:34.528 [conn18] end connection 127.0.0.4:41186 (4 connections now open) m31000| Wed Jan 16 17:26:34.529 [initandlisten] connection accepted from 127.0.0.4:54065 #19 (5 connections now open) m31000| Wed Jan 16 17:26:35.278 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 239, "optime" : { "t" : 1358375195000, "i" : 5582 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358375194000, "i" : 2728 }, "optimeDate" : ISODate("2013-01-16T22:26:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358375195000, "i" : 2462 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:35Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358375194000, "i" : 2728 }, "optimeDate" : ISODate("2013-01-16T22:26:34Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358375191000, "i" : 4064 }, "optimeDate" : ISODate("2013-01-16T22:26:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 225, "optime" : { "t" : 1358375194000, "i" : 358 }, "optimeDate" : ISODate("2013-01-16T22:26:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 225, "optime" : { "t" : 1358375191000, "i" : 4064 }, "optimeDate" : ISODate("2013-01-16T22:26:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358375195000, "i" : 5582 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:37.278 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:37.896 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31001| Wed Jan 16 17:26:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:26:39.278 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 243, "optime" : { "t" : 1358375199000, "i" : 7218 }, "optimeDate" : ISODate("2013-01-16T22:26:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 232, "optime" : { "t" : 1358375198000, "i" : 1194 }, "optimeDate" : ISODate("2013-01-16T22:26:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 232, "optime" : { "t" : 1358375195000, "i" : 5582 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:39Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 242, "optime" : { "t" : 1358375198000, "i" : 1194 }, "optimeDate" : ISODate("2013-01-16T22:26:38Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 230, "optime" : { "t" : 1358375195000, "i" : 5582 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:39Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 229, "optime" : { "t" : 1358375195000, "i" : 5582 }, "optimeDate" : ISODate("2013-01-16T22:26:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 229, "optime" : { "t" : 1358375198000, "i" : 1194 }, "optimeDate" : ISODate("2013-01-16T22:26:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 241, "optime" : { "t" : 1358375199000, "i" : 7218 }, "optimeDate" : ISODate("2013-01-16T22:26:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:41.278 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:43.279 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:43.896 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 248, "optime" : { "t" : 1358375204000, "i" : 1049 }, "optimeDate" : ISODate("2013-01-16T22:26:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358375199000, "i" : 7218 }, "optimeDate" : ISODate("2013-01-16T22:26:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358375199000, "i" : 7218 }, "optimeDate" : ISODate("2013-01-16T22:26:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:44Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 247, "optime" : { "t" : 1358375203000, "i" : 4404 }, "optimeDate" : ISODate("2013-01-16T22:26:43Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 235, "optime" : { "t" : 1358375203000, "i" : 8087 }, "optimeDate" : ISODate("2013-01-16T22:26:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 234, "optime" : { "t" : 1358375199000, "i" : 7218 }, "optimeDate" : ISODate("2013-01-16T22:26:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358375202000, "i" : 374 }, "optimeDate" : ISODate("2013-01-16T22:26:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 246, "optime" : { "t" : 1358375204000, "i" : 1049 }, "optimeDate" : ISODate("2013-01-16T22:26:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:45.279 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:47.279 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:26:47.897 [conn17] end connection 127.0.0.3:37325 (4 connections now open) m31002| Wed Jan 16 17:26:47.897 [initandlisten] connection accepted from 127.0.0.3:43746 #19 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 252, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 241, "optime" : { "t" : 1358375204000, "i" : 1049 }, "optimeDate" : ISODate("2013-01-16T22:26:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 241, "optime" : { "t" : 1358375207000, "i" : 5519 }, "optimeDate" : ISODate("2013-01-16T22:26:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:48Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 251, "optime" : { "t" : 1358375207000, "i" : 6940 }, "optimeDate" : ISODate("2013-01-16T22:26:47Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358375207000, "i" : 7176 }, "optimeDate" : ISODate("2013-01-16T22:26:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 238, "optime" : { "t" : 1358375204000, "i" : 1049 }, "optimeDate" : ISODate("2013-01-16T22:26:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358375204000, "i" : 1049 }, "optimeDate" : ISODate("2013-01-16T22:26:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:47Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 250, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:26:48.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:26:49.280 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:49.898 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.3:31001 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Wed Jan 16 17:26:51.280 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 256, "optime" : { "t" : 1358375212000, "i" : 3881 }, "optimeDate" : ISODate("2013-01-16T22:26:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 245, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 245, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:52Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358375211000, "i" : 5249 }, "optimeDate" : ISODate("2013-01-16T22:26:51Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358375211000, "i" : 5279 }, "optimeDate" : ISODate("2013-01-16T22:26:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 242, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 242, "optime" : { "t" : 1358375208000, "i" : 1870 }, "optimeDate" : ISODate("2013-01-16T22:26:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:51Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358375212000, "i" : 3881 }, "optimeDate" : ISODate("2013-01-16T22:26:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:53.281 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:55.281 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:26:55.898 [rsMgr] not electing self, we are not freshest STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 260, "optime" : { "t" : 1358375216000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-16T22:26:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358375212000, "i" : 3881 }, "optimeDate" : ISODate("2013-01-16T22:26:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358375212000, "i" : 3881 }, "optimeDate" : ISODate("2013-01-16T22:26:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:56Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358375215000, "i" : 3475 }, "optimeDate" : ISODate("2013-01-16T22:26:55Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 247, "optime" : { "t" : 1358375215000, "i" : 3475 }, "optimeDate" : ISODate("2013-01-16T22:26:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:26:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 246, "optime" : { "t" : 1358375216000, "i" : 4610 }, "optimeDate" : ISODate("2013-01-16T22:26:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 246, "optime" : { "t" : 1358375215000, "i" : 3475 }, "optimeDate" : ISODate("2013-01-16T22:26:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358375216000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-16T22:26:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:26:57.281 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:26:59.282 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 264, "optime" : { "t" : 1358375220000, "i" : 6833 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358375216000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-16T22:26:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358375220000, "i" : 1730 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:00Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 263, "optime" : { "t" : 1358375219000, "i" : 1746 }, "optimeDate" : ISODate("2013-01-16T22:26:59Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 251, "optime" : { "t" : 1358375219000, "i" : 1746 }, "optimeDate" : ISODate("2013-01-16T22:26:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 250, "optime" : { "t" : 1358375220000, "i" : 4289 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 250, "optime" : { "t" : 1358375216000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-16T22:26:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:26:59Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 262, "optime" : { "t" : 1358375220000, "i" : 6833 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.4:31002" : [ "127.0.0.3:31001" ], "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:27:01.282 [conn16] end connection 127.0.0.2:48570 (2 connections now open) m31001| Wed Jan 16 17:27:01.282 [initandlisten] connection accepted from 127.0.0.2:41086 #18 (3 connections now open) m31000| Wed Jan 16 17:27:01.283 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:27:02.330 [rsMgr] replSet info electSelf 1 m31002| Wed Jan 16 17:27:02.330 [conn19] replSet info voting yea for 127.0.0.3:31001 (1) m31000| Wed Jan 16 17:27:03.283 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:03.963 [conn8] end connection 127.0.0.3:40699 (4 connections now open) m31001| Wed Jan 16 17:27:03.982 [rsMgr] replSet PRIMARY m31002| Wed Jan 16 17:27:04.025 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state PRIMARY m31002| Wed Jan 16 17:27:04.025 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:04.534 [conn19] end connection 127.0.0.4:54065 (4 connections now open) m31000| Wed Jan 16 17:27:04.534 [initandlisten] connection accepted from 127.0.0.4:41386 #20 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 269, "optime" : { "t" : 1358375225000, "i" : 167 }, "optimeDate" : ISODate("2013-01-16T22:27:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358375220000, "i" : 6833 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358375220000, "i" : 6833 }, "optimeDate" : ISODate("2013-01-16T22:27:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 268, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 256, "optime" : { "t" : 1358375223000, "i" : 479 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 255, "optime" : { "t" : 1358375224000, "i" : 4234 }, "optimeDate" : ISODate("2013-01-16T22:27:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 255, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 267, "optime" : { "t" : 1358375225000, "i" : 167 }, "optimeDate" : ISODate("2013-01-16T22:27:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:27:05.271 [conn18] end connection 127.0.0.2:59087 (3 connections now open) m31000| Wed Jan 16 17:27:05.283 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:05.283 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state PRIMARY m31002| Wed Jan 16 17:27:05.772 [initandlisten] connection accepted from 127.0.0.2:43889 #20 (4 connections now open) m31000| Wed Jan 16 17:27:07.284 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:27:09.026 [conn17] end connection 127.0.0.4:33931 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 273, "optime" : { "t" : 1358375229000, "i" : 2251 }, "optimeDate" : ISODate("2013-01-16T22:27:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 262, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 262, "optime" : { "t" : 1358375228000, "i" : 1362 }, "optimeDate" : ISODate("2013-01-16T22:27:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 272, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358375225000, "i" : 167 }, "optimeDate" : ISODate("2013-01-16T22:27:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 259, "optime" : { "t" : 1358375228000, "i" : 3514 }, "optimeDate" : ISODate("2013-01-16T22:27:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 259, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:07Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358375229000, "i" : 2251 }, "optimeDate" : ISODate("2013-01-16T22:27:09Z"), "self" : true } ], "ok" : 1 } m31000| Wed Jan 16 17:27:09.284 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:27:09.526 [initandlisten] connection accepted from 127.0.0.4:53185 #19 (3 connections now open) m31002| Wed Jan 16 17:27:09.526 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:27:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:27:11.284 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:13.285 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 277, "optime" : { "t" : 1358375233000, "i" : 3570 }, "optimeDate" : ISODate("2013-01-16T22:27:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 266, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358375233000, "i" : 2383 }, "optimeDate" : ISODate("2013-01-16T22:27:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 276, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 264, "optime" : { "t" : 1358375229000, "i" : 2251 }, "optimeDate" : ISODate("2013-01-16T22:27:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:12Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 263, "optime" : { "t" : 1358375232000, "i" : 1743 }, "optimeDate" : ISODate("2013-01-16T22:27:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 263, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 275, "optime" : { "t" : 1358375233000, "i" : 3570 }, "optimeDate" : ISODate("2013-01-16T22:27:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:27:14.527 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:15.285 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:17.285 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 281, "optime" : { "t" : 1358375237000, "i" : 5035 }, "optimeDate" : ISODate("2013-01-16T22:27:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 270, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 270, "optime" : { "t" : 1358375233000, "i" : 3570 }, "optimeDate" : ISODate("2013-01-16T22:27:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 280, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 268, "optime" : { "t" : 1358375233000, "i" : 3570 }, "optimeDate" : ISODate("2013-01-16T22:27:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:17Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 267, "optime" : { "t" : 1358375236000, "i" : 570 }, "optimeDate" : ISODate("2013-01-16T22:27:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 267, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 279, "optime" : { "t" : 1358375237000, "i" : 5035 }, "optimeDate" : ISODate("2013-01-16T22:27:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:27:17.902 [conn19] end connection 127.0.0.3:43746 (3 connections now open) m31002| Wed Jan 16 17:27:17.902 [initandlisten] connection accepted from 127.0.0.3:41910 #21 (4 connections now open) m31000| Wed Jan 16 17:27:19.286 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:19.528 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:27:20.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:27:21.286 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 285, "optime" : { "t" : 1358375241000, "i" : 6717 }, "optimeDate" : ISODate("2013-01-16T22:27:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 274, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 274, "optime" : { "t" : 1358375240000, "i" : 1515 }, "optimeDate" : ISODate("2013-01-16T22:27:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 284, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 272, "optime" : { "t" : 1358375237000, "i" : 5035 }, "optimeDate" : ISODate("2013-01-16T22:27:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 271, "optime" : { "t" : 1358375237000, "i" : 5035 }, "optimeDate" : ISODate("2013-01-16T22:27:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 271, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:19Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 283, "optime" : { "t" : 1358375241000, "i" : 6717 }, "optimeDate" : ISODate("2013-01-16T22:27:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:27:23.286 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:24.528 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:25.287 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 289, "optime" : { "t" : 1358375245000, "i" : 7910 }, "optimeDate" : ISODate("2013-01-16T22:27:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 278, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 278, "optime" : { "t" : 1358375245000, "i" : 7044 }, "optimeDate" : ISODate("2013-01-16T22:27:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 288, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358375241000, "i" : 6717 }, "optimeDate" : ISODate("2013-01-16T22:27:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:24Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 275, "optime" : { "t" : 1358375241000, "i" : 6717 }, "optimeDate" : ISODate("2013-01-16T22:27:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 275, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358375245000, "i" : 7910 }, "optimeDate" : ISODate("2013-01-16T22:27:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:27:27.287 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:29.287 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:29.529 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 294, "optime" : { "t" : 1358375250000, "i" : 324 }, "optimeDate" : ISODate("2013-01-16T22:27:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 283, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 283, "optime" : { "t" : 1358375245000, "i" : 7910 }, "optimeDate" : ISODate("2013-01-16T22:27:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:26:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 293, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358375249000, "i" : 6960 }, "optimeDate" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 280, "optime" : { "t" : 1358375245000, "i" : 7910 }, "optimeDate" : ISODate("2013-01-16T22:27:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 280, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:29Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358375249000, "i" : 8444 }, "optimeDate" : ISODate("2013-01-16T22:27:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:27:31.287 [conn18] end connection 127.0.0.2:41086 (2 connections now open) m31001| Wed Jan 16 17:27:31.288 [initandlisten] connection accepted from 127.0.0.2:34576 #20 (3 connections now open) m31000| Wed Jan 16 17:27:31.288 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:33.288 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 298, "optime" : { "t" : 1358375254000, "i" : 1004 }, "optimeDate" : ISODate("2013-01-16T22:27:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 287, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358375253000, "i" : 1479 }, "optimeDate" : ISODate("2013-01-16T22:27:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 297, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 285, "optime" : { "t" : 1358375253000, "i" : 7349 }, "optimeDate" : ISODate("2013-01-16T22:27:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 284, "optime" : { "t" : 1358375250000, "i" : 324 }, "optimeDate" : ISODate("2013-01-16T22:27:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 284, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:33Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 296, "optime" : { "t" : 1358375254000, "i" : 973 }, "optimeDate" : ISODate("2013-01-16T22:27:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:27:34.530 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:34.539 [conn20] end connection 127.0.0.4:41386 (4 connections now open) m31000| Wed Jan 16 17:27:34.539 [initandlisten] connection accepted from 127.0.0.4:49387 #21 (5 connections now open) m31000| Wed Jan 16 17:27:35.289 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:37.289 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 302, "optime" : { "t" : 1358375258000, "i" : 2136 }, "optimeDate" : ISODate("2013-01-16T22:27:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 291, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 291, "optime" : { "t" : 1358375254000, "i" : 1004 }, "optimeDate" : ISODate("2013-01-16T22:27:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 301, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 289, "optime" : { "t" : 1358375257000, "i" : 6910 }, "optimeDate" : ISODate("2013-01-16T22:27:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 288, "optime" : { "t" : 1358375254000, "i" : 1004 }, "optimeDate" : ISODate("2013-01-16T22:27:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 288, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:37Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 300, "optime" : { "t" : 1358375258000, "i" : 2105 }, "optimeDate" : ISODate("2013-01-16T22:27:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:27:39.289 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:39.530 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:41.290 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 306, "optime" : { "t" : 1358375262000, "i" : 3141 }, "optimeDate" : ISODate("2013-01-16T22:27:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 295, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 295, "optime" : { "t" : 1358375258000, "i" : 2136 }, "optimeDate" : ISODate("2013-01-16T22:27:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 305, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 293, "optime" : { "t" : 1358375261000, "i" : 5596 }, "optimeDate" : ISODate("2013-01-16T22:27:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 292, "optime" : { "t" : 1358375258000, "i" : 2136 }, "optimeDate" : ISODate("2013-01-16T22:27:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 292, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 304, "optime" : { "t" : 1358375262000, "i" : 2736 }, "optimeDate" : ISODate("2013-01-16T22:27:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:27:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:27:43.276 [conn20] end connection 127.0.0.2:43889 (3 connections now open) m31000| Wed Jan 16 17:27:43.290 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:43.777 [initandlisten] connection accepted from 127.0.0.2:50448 #22 (4 connections now open) m31002| Wed Jan 16 17:27:44.531 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:45.290 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 310, "optime" : { "t" : 1358375266000, "i" : 4687 }, "optimeDate" : ISODate("2013-01-16T22:27:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 299, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 299, "optime" : { "t" : 1358375266000, "i" : 2088 }, "optimeDate" : ISODate("2013-01-16T22:27:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 309, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358375265000, "i" : 4663 }, "optimeDate" : ISODate("2013-01-16T22:27:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 296, "optime" : { "t" : 1358375262000, "i" : 3141 }, "optimeDate" : ISODate("2013-01-16T22:27:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 296, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:45Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 308, "optime" : { "t" : 1358375266000, "i" : 4687 }, "optimeDate" : ISODate("2013-01-16T22:27:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:27:47.031 [conn19] end connection 127.0.0.4:53185 (2 connections now open) m31000| Wed Jan 16 17:27:47.291 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:27:47.531 [initandlisten] connection accepted from 127.0.0.4:43173 #21 (3 connections now open) m31002| Wed Jan 16 17:27:47.907 [conn21] end connection 127.0.0.3:41910 (3 connections now open) m31002| Wed Jan 16 17:27:47.907 [initandlisten] connection accepted from 127.0.0.3:35994 #23 (4 connections now open) m31000| Wed Jan 16 17:27:49.291 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:27:50.032 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 314, "optime" : { "t" : 1358375270000, "i" : 5929 }, "optimeDate" : ISODate("2013-01-16T22:27:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 303, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 303, "optime" : { "t" : 1358375266000, "i" : 4687 }, "optimeDate" : ISODate("2013-01-16T22:27:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 313, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 301, "optime" : { "t" : 1358375269000, "i" : 2832 }, "optimeDate" : ISODate("2013-01-16T22:27:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 300, "optime" : { "t" : 1358375270000, "i" : 4504 }, "optimeDate" : ISODate("2013-01-16T22:27:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 300, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358375270000, "i" : 5929 }, "optimeDate" : ISODate("2013-01-16T22:27:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:27:51.292 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:27:52.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:27:53.292 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 318, "optime" : { "t" : 1358375274000, "i" : 7355 }, "optimeDate" : ISODate("2013-01-16T22:27:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 307, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358375273000, "i" : 185 }, "optimeDate" : ISODate("2013-01-16T22:27:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 317, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 305, "optime" : { "t" : 1358375273000, "i" : 1424 }, "optimeDate" : ISODate("2013-01-16T22:27:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 304, "optime" : { "t" : 1358375274000, "i" : 4511 }, "optimeDate" : ISODate("2013-01-16T22:27:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 304, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:53Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 316, "optime" : { "t" : 1358375274000, "i" : 7355 }, "optimeDate" : ISODate("2013-01-16T22:27:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:27:55.033 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:27:55.292 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:27:57.293 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 322, "optime" : { "t" : 1358375278000, "i" : 8835 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 311, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311, "optime" : { "t" : 1358375278000, "i" : 7186 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 321, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 309, "optime" : { "t" : 1358375277000, "i" : 260 }, "optimeDate" : ISODate("2013-01-16T22:27:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:27:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 308, "optime" : { "t" : 1358375278000, "i" : 4775 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 308, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:27:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:27:57Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 320, "optime" : { "t" : 1358375278000, "i" : 8835 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:27:59.293 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:00.034 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:28:01.293 [conn20] end connection 127.0.0.2:34576 (2 connections now open) m31001| Wed Jan 16 17:28:01.293 [initandlisten] connection accepted from 127.0.0.2:43324 #22 (3 connections now open) m31000| Wed Jan 16 17:28:01.294 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:02.924 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:226662 113ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 327, "optime" : { "t" : 1358375283000, "i" : 791 }, "optimeDate" : ISODate("2013-01-16T22:28:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 316, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 316, "optime" : { "t" : 1358375278000, "i" : 8835 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 326, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 314, "optime" : { "t" : 1358375278000, "i" : 8835 }, "optimeDate" : ISODate("2013-01-16T22:27:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 313, "optime" : { "t" : 1358375282000, "i" : 4377 }, "optimeDate" : ISODate("2013-01-16T22:28:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 313, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 325, "optime" : { "t" : 1358375283000, "i" : 791 }, "optimeDate" : ISODate("2013-01-16T22:28:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:03.294 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:04.544 [conn21] end connection 127.0.0.4:49387 (4 connections now open) m31000| Wed Jan 16 17:28:04.544 [initandlisten] connection accepted from 127.0.0.4:46438 #22 (5 connections now open) m31002| Wed Jan 16 17:28:05.034 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:28:05.294 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 331, "optime" : { "t" : 1358375287000, "i" : 2046 }, "optimeDate" : ISODate("2013-01-16T22:28:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 320, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 320, "optime" : { "t" : 1358375286000, "i" : 1052 }, "optimeDate" : ISODate("2013-01-16T22:28:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 330, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358375283000, "i" : 791 }, "optimeDate" : ISODate("2013-01-16T22:28:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 317, "optime" : { "t" : 1358375286000, "i" : 3564 }, "optimeDate" : ISODate("2013-01-16T22:28:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 317, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:05Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 329, "optime" : { "t" : 1358375287000, "i" : 1747 }, "optimeDate" : ISODate("2013-01-16T22:28:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:07.295 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:09.295 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:10.035 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:28:11.295 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 335, "optime" : { "t" : 1358375291000, "i" : 3969 }, "optimeDate" : ISODate("2013-01-16T22:28:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 324, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 324, "optime" : { "t" : 1358375291000, "i" : 2437 }, "optimeDate" : ISODate("2013-01-16T22:28:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 334, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 322, "optime" : { "t" : 1358375287000, "i" : 2046 }, "optimeDate" : ISODate("2013-01-16T22:28:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 321, "optime" : { "t" : 1358375290000, "i" : 2575 }, "optimeDate" : ISODate("2013-01-16T22:28:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 321, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358375291000, "i" : 3763 }, "optimeDate" : ISODate("2013-01-16T22:28:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:13.296 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:28:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:28:15.035 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:28:15.296 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 339, "optime" : { "t" : 1358375295000, "i" : 5171 }, "optimeDate" : ISODate("2013-01-16T22:28:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 328, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358375291000, "i" : 3969 }, "optimeDate" : ISODate("2013-01-16T22:28:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 338, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 326, "optime" : { "t" : 1358375291000, "i" : 3969 }, "optimeDate" : ISODate("2013-01-16T22:28:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:15Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 325, "optime" : { "t" : 1358375294000, "i" : 1058 }, "optimeDate" : ISODate("2013-01-16T22:28:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 325, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 337, "optime" : { "t" : 1358375295000, "i" : 5171 }, "optimeDate" : ISODate("2013-01-16T22:28:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:17.296 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:17.913 [conn23] end connection 127.0.0.3:35994 (3 connections now open) m31002| Wed Jan 16 17:28:17.913 [initandlisten] connection accepted from 127.0.0.3:36141 #24 (4 connections now open) m31000| Wed Jan 16 17:28:19.297 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 343, "optime" : { "t" : 1358375299000, "i" : 5628 }, "optimeDate" : ISODate("2013-01-16T22:28:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 332, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 332, "optime" : { "t" : 1358375298000, "i" : 2242 }, "optimeDate" : ISODate("2013-01-16T22:28:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 342, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 330, "optime" : { "t" : 1358375295000, "i" : 5171 }, "optimeDate" : ISODate("2013-01-16T22:28:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 329, "optime" : { "t" : 1358375298000, "i" : 88 }, "optimeDate" : ISODate("2013-01-16T22:28:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 329, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:17Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 341, "optime" : { "t" : 1358375299000, "i" : 5628 }, "optimeDate" : ISODate("2013-01-16T22:28:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:28:20.036 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:28:21.282 [conn22] end connection 127.0.0.2:50448 (3 connections now open) m31000| Wed Jan 16 17:28:21.297 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:21.782 [initandlisten] connection accepted from 127.0.0.2:60633 #25 (4 connections now open) m31000| Wed Jan 16 17:28:23.297 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 347, "optime" : { "t" : 1358375303000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T22:28:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 336, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 336, "optime" : { "t" : 1358375299000, "i" : 5628 }, "optimeDate" : ISODate("2013-01-16T22:28:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 346, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 334, "optime" : { "t" : 1358375299000, "i" : 5628 }, "optimeDate" : ISODate("2013-01-16T22:28:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 333, "optime" : { "t" : 1358375299000, "i" : 5628 }, "optimeDate" : ISODate("2013-01-16T22:28:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 333, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 345, "optime" : { "t" : 1358375303000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T22:28:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:28:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:28:25.037 [conn21] end connection 127.0.0.4:43173 (2 connections now open) m31000| Wed Jan 16 17:28:25.298 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:28:25.537 [initandlisten] connection accepted from 127.0.0.4:46695 #23 (3 connections now open) m31002| Wed Jan 16 17:28:25.537 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:28:27.298 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:28:28.019 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293711 146ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 352, "optime" : { "t" : 1358375308000, "i" : 527 }, "optimeDate" : ISODate("2013-01-16T22:28:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 341, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 341, "optime" : { "t" : 1358375303000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T22:28:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 351, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 339, "optime" : { "t" : 1358375307000, "i" : 6453 }, "optimeDate" : ISODate("2013-01-16T22:28:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 338, "optime" : { "t" : 1358375303000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T22:28:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 338, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 350, "optime" : { "t" : 1358375307000, "i" : 7980 }, "optimeDate" : ISODate("2013-01-16T22:28:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:29.299 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:30.538 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:28:31.299 [conn22] end connection 127.0.0.2:43324 (2 connections now open) m31001| Wed Jan 16 17:28:31.299 [initandlisten] connection accepted from 127.0.0.2:47243 #24 (4 connections now open) m31000| Wed Jan 16 17:28:31.299 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 356, "optime" : { "t" : 1358375312000, "i" : 1812 }, "optimeDate" : ISODate("2013-01-16T22:28:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 345, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 345, "optime" : { "t" : 1358375311000, "i" : 6341 }, "optimeDate" : ISODate("2013-01-16T22:28:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 355, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 343, "optime" : { "t" : 1358375311000, "i" : 7445 }, "optimeDate" : ISODate("2013-01-16T22:28:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 342, "optime" : { "t" : 1358375308000, "i" : 527 }, "optimeDate" : ISODate("2013-01-16T22:28:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 342, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:31Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358375312000, "i" : 1812 }, "optimeDate" : ISODate("2013-01-16T22:28:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:33.299 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:34.550 [conn22] end connection 127.0.0.4:46438 (4 connections now open) m31000| Wed Jan 16 17:28:34.550 [initandlisten] connection accepted from 127.0.0.4:32940 #23 (5 connections now open) m31000| Wed Jan 16 17:28:35.300 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:35.538 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 360, "optime" : { "t" : 1358375316000, "i" : 3532 }, "optimeDate" : ISODate("2013-01-16T22:28:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 349, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358375312000, "i" : 1812 }, "optimeDate" : ISODate("2013-01-16T22:28:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 359, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 347, "optime" : { "t" : 1358375315000, "i" : 5988 }, "optimeDate" : ISODate("2013-01-16T22:28:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 346, "optime" : { "t" : 1358375312000, "i" : 1812 }, "optimeDate" : ISODate("2013-01-16T22:28:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 346, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:35Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 358, "optime" : { "t" : 1358375316000, "i" : 3532 }, "optimeDate" : ISODate("2013-01-16T22:28:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:37.300 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:39.300 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 17:28:40.539 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 364, "optime" : { "t" : 1358375320000, "i" : 4721 }, "optimeDate" : ISODate("2013-01-16T22:28:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 353, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 353, "optime" : { "t" : 1358375316000, "i" : 3532 }, "optimeDate" : ISODate("2013-01-16T22:28:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 363, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 351, "optime" : { "t" : 1358375319000, "i" : 4113 }, "optimeDate" : ISODate("2013-01-16T22:28:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 350, "optime" : { "t" : 1358375316000, "i" : 3532 }, "optimeDate" : ISODate("2013-01-16T22:28:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 350, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 362, "optime" : { "t" : 1358375320000, "i" : 4682 }, "optimeDate" : ISODate("2013-01-16T22:28:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:41.301 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:43.301 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 368, "optime" : { "t" : 1358375324000, "i" : 5887 }, "optimeDate" : ISODate("2013-01-16T22:28:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 357, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 357, "optime" : { "t" : 1358375324000, "i" : 1833 }, "optimeDate" : ISODate("2013-01-16T22:28:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 367, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 355, "optime" : { "t" : 1358375323000, "i" : 3054 }, "optimeDate" : ISODate("2013-01-16T22:28:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 354, "optime" : { "t" : 1358375324000, "i" : 4859 }, "optimeDate" : ISODate("2013-01-16T22:28:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 354, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:43Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 366, "optime" : { "t" : 1358375324000, "i" : 5364 }, "optimeDate" : ISODate("2013-01-16T22:28:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:45.302 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:45.540 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:28:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:28:47.302 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:47.918 [conn24] end connection 127.0.0.3:36141 (3 connections now open) m31002| Wed Jan 16 17:28:47.918 [initandlisten] connection accepted from 127.0.0.3:58539 #26 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 372, "optime" : { "t" : 1358375328000, "i" : 6979 }, "optimeDate" : ISODate("2013-01-16T22:28:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 361, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 361, "optime" : { "t" : 1358375324000, "i" : 5887 }, "optimeDate" : ISODate("2013-01-16T22:28:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 371, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 359, "optime" : { "t" : 1358375327000, "i" : 1814 }, "optimeDate" : ISODate("2013-01-16T22:28:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 358, "optime" : { "t" : 1358375328000, "i" : 4701 }, "optimeDate" : ISODate("2013-01-16T22:28:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 358, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 370, "optime" : { "t" : 1358375328000, "i" : 6979 }, "optimeDate" : ISODate("2013-01-16T22:28:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:49.302 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:50.541 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:28:51.303 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:28:53.021 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:462199 231ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 377, "optime" : { "t" : 1358375333000, "i" : 1592 }, "optimeDate" : ISODate("2013-01-16T22:28:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 366, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 366, "optime" : { "t" : 1358375328000, "i" : 6979 }, "optimeDate" : ISODate("2013-01-16T22:28:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 376, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 364, "optime" : { "t" : 1358375331000, "i" : 378 }, "optimeDate" : ISODate("2013-01-16T22:28:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 363, "optime" : { "t" : 1358375332000, "i" : 4726 }, "optimeDate" : ISODate("2013-01-16T22:28:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 363, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 375, "optime" : { "t" : 1358375333000, "i" : 1592 }, "optimeDate" : ISODate("2013-01-16T22:28:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:53.303 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:28:55.304 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:55.541 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:28:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 381, "optime" : { "t" : 1358375337000, "i" : 1822 }, "optimeDate" : ISODate("2013-01-16T22:28:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 370, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 370, "optime" : { "t" : 1358375336000, "i" : 6673 }, "optimeDate" : ISODate("2013-01-16T22:28:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 380, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 368, "optime" : { "t" : 1358375333000, "i" : 1592 }, "optimeDate" : ISODate("2013-01-16T22:28:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:28:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 367, "optime" : { "t" : 1358375336000, "i" : 3684 }, "optimeDate" : ISODate("2013-01-16T22:28:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:56Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 367, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:28:55Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 379, "optime" : { "t" : 1358375337000, "i" : 1351 }, "optimeDate" : ISODate("2013-01-16T22:28:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:28:57.304 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:59.287 [conn25] end connection 127.0.0.2:60633 (3 connections now open) m31000| Wed Jan 16 17:28:59.305 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:28:59.787 [initandlisten] connection accepted from 127.0.0.2:37286 #27 (4 connections now open) m31002| Wed Jan 16 17:29:00.542 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:29:01.305 [conn24] end connection 127.0.0.2:47243 (2 connections now open) m31001| Wed Jan 16 17:29:01.305 [initandlisten] connection accepted from 127.0.0.2:60454 #25 (3 connections now open) m31000| Wed Jan 16 17:29:01.305 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 385, "optime" : { "t" : 1358375341000, "i" : 2679 }, "optimeDate" : ISODate("2013-01-16T22:29:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 374, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 374, "optime" : { "t" : 1358375337000, "i" : 1822 }, "optimeDate" : ISODate("2013-01-16T22:28:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 384, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 372, "optime" : { "t" : 1358375337000, "i" : 1822 }, "optimeDate" : ISODate("2013-01-16T22:28:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 371, "optime" : { "t" : 1358375340000, "i" : 3276 }, "optimeDate" : ISODate("2013-01-16T22:29:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 371, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 383, "optime" : { "t" : 1358375341000, "i" : 2679 }, "optimeDate" : ISODate("2013-01-16T22:29:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:29:03.042 [conn23] end connection 127.0.0.4:46695 (2 connections now open) m31000| Wed Jan 16 17:29:03.305 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:29:03.542 [initandlisten] connection accepted from 127.0.0.4:56259 #26 (3 connections now open) m31000| Wed Jan 16 17:29:04.554 [conn23] end connection 127.0.0.4:32940 (4 connections now open) m31000| Wed Jan 16 17:29:04.554 [initandlisten] connection accepted from 127.0.0.4:38282 #24 (5 connections now open) m31000| Wed Jan 16 17:29:05.306 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 389, "optime" : { "t" : 1358375345000, "i" : 4051 }, "optimeDate" : ISODate("2013-01-16T22:29:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 378, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 378, "optime" : { "t" : 1358375344000, "i" : 4061 }, "optimeDate" : ISODate("2013-01-16T22:29:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:28:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 388, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 376, "optime" : { "t" : 1358375341000, "i" : 2679 }, "optimeDate" : ISODate("2013-01-16T22:29:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 375, "optime" : { "t" : 1358375344000, "i" : 1878 }, "optimeDate" : ISODate("2013-01-16T22:29:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 375, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:03Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 387, "optime" : { "t" : 1358375345000, "i" : 4051 }, "optimeDate" : ISODate("2013-01-16T22:29:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:29:06.043 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:07.306 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:09.306 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 393, "optime" : { "t" : 1358375349000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:29:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 382, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 382, "optime" : { "t" : 1358375345000, "i" : 4051 }, "optimeDate" : ISODate("2013-01-16T22:29:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 392, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 380, "optime" : { "t" : 1358375345000, "i" : 4051 }, "optimeDate" : ISODate("2013-01-16T22:29:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 379, "optime" : { "t" : 1358375348000, "i" : 222 }, "optimeDate" : ISODate("2013-01-16T22:29:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 379, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 391, "optime" : { "t" : 1358375349000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:29:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:29:11.043 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:11.307 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:12.943 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:381715 190ms m31000| Wed Jan 16 17:29:13.307 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 397, "optime" : { "t" : 1358375353000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-16T22:29:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 386, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 386, "optime" : { "t" : 1358375349000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:29:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 396, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 384, "optime" : { "t" : 1358375349000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:29:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 383, "optime" : { "t" : 1358375349000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:29:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 383, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 395, "optime" : { "t" : 1358375353000, "i" : 7761 }, "optimeDate" : ISODate("2013-01-16T22:29:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:15.307 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:16.044 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:17.308 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 17:29:17.924 [conn26] end connection 127.0.0.3:58539 (3 connections now open) m31002| Wed Jan 16 17:29:17.924 [initandlisten] connection accepted from 127.0.0.3:50101 #28 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 401, "optime" : { "t" : 1358375357000, "i" : 7689 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 390, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 390, "optime" : { "t" : 1358375357000, "i" : 1630 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 400, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 388, "optime" : { "t" : 1358375357000, "i" : 7378 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 387, "optime" : { "t" : 1358375353000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-16T22:29:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 387, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:17Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 399, "optime" : { "t" : 1358375357000, "i" : 7689 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:29:18.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:29:19.308 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:21.045 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:21.308 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 406, "optime" : { "t" : 1358375362000, "i" : 1152 }, "optimeDate" : ISODate("2013-01-16T22:29:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 395, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 395, "optime" : { "t" : 1358375357000, "i" : 7689 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 405, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 393, "optime" : { "t" : 1358375361000, "i" : 8329 }, "optimeDate" : ISODate("2013-01-16T22:29:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 392, "optime" : { "t" : 1358375357000, "i" : 7689 }, "optimeDate" : ISODate("2013-01-16T22:29:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 392, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:21Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 404, "optime" : { "t" : 1358375362000, "i" : 1152 }, "optimeDate" : ISODate("2013-01-16T22:29:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:23.309 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:25.309 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:26.045 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 410, "optime" : { "t" : 1358375366000, "i" : 2470 }, "optimeDate" : ISODate("2013-01-16T22:29:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 399, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 399, "optime" : { "t" : 1358375362000, "i" : 1152 }, "optimeDate" : ISODate("2013-01-16T22:29:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 409, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 397, "optime" : { "t" : 1358375365000, "i" : 6559 }, "optimeDate" : ISODate("2013-01-16T22:29:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 396, "optime" : { "t" : 1358375362000, "i" : 1152 }, "optimeDate" : ISODate("2013-01-16T22:29:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 396, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 408, "optime" : { "t" : 1358375366000, "i" : 2296 }, "optimeDate" : ISODate("2013-01-16T22:29:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:27.309 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:29:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:29:29.310 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 414, "optime" : { "t" : 1358375370000, "i" : 4109 }, "optimeDate" : ISODate("2013-01-16T22:29:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 403, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 403, "optime" : { "t" : 1358375369000, "i" : 4209 }, "optimeDate" : ISODate("2013-01-16T22:29:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 413, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 401, "optime" : { "t" : 1358375369000, "i" : 5453 }, "optimeDate" : ISODate("2013-01-16T22:29:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 400, "optime" : { "t" : 1358375366000, "i" : 2470 }, "optimeDate" : ISODate("2013-01-16T22:29:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 400, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:29Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 412, "optime" : { "t" : 1358375370000, "i" : 3948 }, "optimeDate" : ISODate("2013-01-16T22:29:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:29:31.046 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:29:31.310 [conn25] end connection 127.0.0.2:60454 (2 connections now open) m31001| Wed Jan 16 17:29:31.310 [initandlisten] connection accepted from 127.0.0.2:51227 #27 (3 connections now open) m31000| Wed Jan 16 17:29:31.311 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:33.311 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 418, "optime" : { "t" : 1358375374000, "i" : 5278 }, "optimeDate" : ISODate("2013-01-16T22:29:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 407, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 407, "optime" : { "t" : 1358375370000, "i" : 4109 }, "optimeDate" : ISODate("2013-01-16T22:29:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 417, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 405, "optime" : { "t" : 1358375373000, "i" : 4399 }, "optimeDate" : ISODate("2013-01-16T22:29:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 404, "optime" : { "t" : 1358375370000, "i" : 4109 }, "optimeDate" : ISODate("2013-01-16T22:29:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 404, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:33Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 416, "optime" : { "t" : 1358375374000, "i" : 5278 }, "optimeDate" : ISODate("2013-01-16T22:29:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:34.560 [conn24] end connection 127.0.0.4:38282 (4 connections now open) m31000| Wed Jan 16 17:29:34.560 [initandlisten] connection accepted from 127.0.0.4:49619 #25 (5 connections now open) m31000| Wed Jan 16 17:29:35.311 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:36.047 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:29:37.293 [conn27] end connection 127.0.0.2:37286 (3 connections now open) m31000| Wed Jan 16 17:29:37.312 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:37.793 [initandlisten] connection accepted from 127.0.0.2:55193 #29 (4 connections now open) m31000| Wed Jan 16 17:29:38.372 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293961 146ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 422, "optime" : { "t" : 1358375378000, "i" : 5681 }, "optimeDate" : ISODate("2013-01-16T22:29:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 411, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 411, "optime" : { "t" : 1358375377000, "i" : 1594 }, "optimeDate" : ISODate("2013-01-16T22:29:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 421, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 409, "optime" : { "t" : 1358375377000, "i" : 3082 }, "optimeDate" : ISODate("2013-01-16T22:29:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 408, "optime" : { "t" : 1358375374000, "i" : 5278 }, "optimeDate" : ISODate("2013-01-16T22:29:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 408, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:37Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 420, "optime" : { "t" : 1358375378000, "i" : 5250 }, "optimeDate" : ISODate("2013-01-16T22:29:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:39.312 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:29:41.047 [conn26] end connection 127.0.0.4:56259 (2 connections now open) m31000| Wed Jan 16 17:29:41.313 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:29:41.547 [initandlisten] connection accepted from 127.0.0.4:55700 #28 (3 connections now open) m31002| Wed Jan 16 17:29:41.548 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 426, "optime" : { "t" : 1358375382000, "i" : 6241 }, "optimeDate" : ISODate("2013-01-16T22:29:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 415, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 415, "optime" : { "t" : 1358375378000, "i" : 5681 }, "optimeDate" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 425, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 413, "optime" : { "t" : 1358375381000, "i" : 2679 }, "optimeDate" : ISODate("2013-01-16T22:29:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 412, "optime" : { "t" : 1358375382000, "i" : 4475 }, "optimeDate" : ISODate("2013-01-16T22:29:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 412, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:41Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 424, "optime" : { "t" : 1358375382000, "i" : 6241 }, "optimeDate" : ISODate("2013-01-16T22:29:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:43.313 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:45.313 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:46.548 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 430, "optime" : { "t" : 1358375386000, "i" : 7858 }, "optimeDate" : ISODate("2013-01-16T22:29:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 419, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 419, "optime" : { "t" : 1358375382000, "i" : 6241 }, "optimeDate" : ISODate("2013-01-16T22:29:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 429, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 417, "optime" : { "t" : 1358375385000, "i" : 1318 }, "optimeDate" : ISODate("2013-01-16T22:29:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 416, "optime" : { "t" : 1358375386000, "i" : 4563 }, "optimeDate" : ISODate("2013-01-16T22:29:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 416, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 428, "optime" : { "t" : 1358375386000, "i" : 7858 }, "optimeDate" : ISODate("2013-01-16T22:29:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:47.314 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:47.928 [conn28] end connection 127.0.0.3:50101 (3 connections now open) m31002| Wed Jan 16 17:29:47.929 [initandlisten] connection accepted from 127.0.0.3:59959 #30 (4 connections now open) m31000| Wed Jan 16 17:29:49.314 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:29:50.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 434, "optime" : { "t" : 1358375390000, "i" : 8899 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 423, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 423, "optime" : { "t" : 1358375390000, "i" : 1900 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 433, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 421, "optime" : { "t" : 1358375389000, "i" : 273 }, "optimeDate" : ISODate("2013-01-16T22:29:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 420, "optime" : { "t" : 1358375390000, "i" : 4817 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 420, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:49Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 432, "optime" : { "t" : 1358375390000, "i" : 8218 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:51.314 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:51.549 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:53.315 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 439, "optime" : { "t" : 1358375395000, "i" : 1461 }, "optimeDate" : ISODate("2013-01-16T22:29:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 428, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 428, "optime" : { "t" : 1358375390000, "i" : 8899 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 438, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 426, "optime" : { "t" : 1358375390000, "i" : 8899 }, "optimeDate" : ISODate("2013-01-16T22:29:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 425, "optime" : { "t" : 1358375394000, "i" : 4510 }, "optimeDate" : ISODate("2013-01-16T22:29:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 425, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 437, "optime" : { "t" : 1358375395000, "i" : 1461 }, "optimeDate" : ISODate("2013-01-16T22:29:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:29:55.315 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:29:56.550 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:29:57.316 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:29:59.316 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 443, "optime" : { "t" : 1358375399000, "i" : 3398 }, "optimeDate" : ISODate("2013-01-16T22:29:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 432, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 432, "optime" : { "t" : 1358375395000, "i" : 1461 }, "optimeDate" : ISODate("2013-01-16T22:29:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 442, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 430, "optime" : { "t" : 1358375395000, "i" : 1461 }, "optimeDate" : ISODate("2013-01-16T22:29:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:29:59Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:29:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 429, "optime" : { "t" : 1358375398000, "i" : 2589 }, "optimeDate" : ISODate("2013-01-16T22:29:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 429, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 441, "optime" : { "t" : 1358375399000, "i" : 3398 }, "optimeDate" : ISODate("2013-01-16T22:29:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:30:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:30:01.316 [conn27] end connection 127.0.0.2:51227 (2 connections now open) m31001| Wed Jan 16 17:30:01.316 [initandlisten] connection accepted from 127.0.0.2:46598 #29 (3 connections now open) m31000| Wed Jan 16 17:30:01.317 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:01.550 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:03.049 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:527088 263ms m31000| Wed Jan 16 17:30:03.317 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 447, "optime" : { "t" : 1358375403000, "i" : 7134 }, "optimeDate" : ISODate("2013-01-16T22:30:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 436, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 436, "optime" : { "t" : 1358375402000, "i" : 2864 }, "optimeDate" : ISODate("2013-01-16T22:30:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 446, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 434, "optime" : { "t" : 1358375399000, "i" : 3398 }, "optimeDate" : ISODate("2013-01-16T22:29:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 433, "optime" : { "t" : 1358375402000, "i" : 1107 }, "optimeDate" : ISODate("2013-01-16T22:30:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 433, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:01Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 445, "optime" : { "t" : 1358375403000, "i" : 6891 }, "optimeDate" : ISODate("2013-01-16T22:30:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:04.565 [conn25] end connection 127.0.0.4:49619 (4 connections now open) m31000| Wed Jan 16 17:30:04.565 [initandlisten] connection accepted from 127.0.0.4:50921 #26 (5 connections now open) m31000| Wed Jan 16 17:30:05.317 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:06.551 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:07.317 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 451, "optime" : { "t" : 1358375407000, "i" : 5378 }, "optimeDate" : ISODate("2013-01-16T22:30:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 440, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 440, "optime" : { "t" : 1358375403000, "i" : 7134 }, "optimeDate" : ISODate("2013-01-16T22:30:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:29:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 450, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 438, "optime" : { "t" : 1358375403000, "i" : 7134 }, "optimeDate" : ISODate("2013-01-16T22:30:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 437, "optime" : { "t" : 1358375406000, "i" : 492 }, "optimeDate" : ISODate("2013-01-16T22:30:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 437, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 449, "optime" : { "t" : 1358375407000, "i" : 5378 }, "optimeDate" : ISODate("2013-01-16T22:30:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:09.318 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:30:11.318 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:11.551 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 455, "optime" : { "t" : 1358375411000, "i" : 6629 }, "optimeDate" : ISODate("2013-01-16T22:30:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 444, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 444, "optime" : { "t" : 1358375407000, "i" : 5378 }, "optimeDate" : ISODate("2013-01-16T22:30:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 454, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 442, "optime" : { "t" : 1358375407000, "i" : 5378 }, "optimeDate" : ISODate("2013-01-16T22:30:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 441, "optime" : { "t" : 1358375407000, "i" : 5378 }, "optimeDate" : ISODate("2013-01-16T22:30:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 441, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 453, "optime" : { "t" : 1358375411000, "i" : 6629 }, "optimeDate" : ISODate("2013-01-16T22:30:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:13.318 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:15.298 [conn29] end connection 127.0.0.2:55193 (3 connections now open) m31000| Wed Jan 16 17:30:15.319 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:15.798 [initandlisten] connection accepted from 127.0.0.2:39672 #31 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 459, "optime" : { "t" : 1358375415000, "i" : 7924 }, "optimeDate" : ISODate("2013-01-16T22:30:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 448, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 448, "optime" : { "t" : 1358375415000, "i" : 7107 }, "optimeDate" : ISODate("2013-01-16T22:30:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 458, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 446, "optime" : { "t" : 1358375411000, "i" : 6629 }, "optimeDate" : ISODate("2013-01-16T22:30:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 445, "optime" : { "t" : 1358375411000, "i" : 6629 }, "optimeDate" : ISODate("2013-01-16T22:30:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 445, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 457, "optime" : { "t" : 1358375415000, "i" : 7829 }, "optimeDate" : ISODate("2013-01-16T22:30:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:30:16.552 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:17.319 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:17.934 [conn30] end connection 127.0.0.3:59959 (3 connections now open) m31002| Wed Jan 16 17:30:17.934 [initandlisten] connection accepted from 127.0.0.3:34309 #32 (4 connections now open) m31001| Wed Jan 16 17:30:19.052 [conn28] end connection 127.0.0.4:55700 (2 connections now open) m31000| Wed Jan 16 17:30:19.319 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:30:19.552 [initandlisten] connection accepted from 127.0.0.4:57185 #30 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 464, "optime" : { "t" : 1358375420000, "i" : 593 }, "optimeDate" : ISODate("2013-01-16T22:30:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 453, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 453, "optime" : { "t" : 1358375415000, "i" : 7924 }, "optimeDate" : ISODate("2013-01-16T22:30:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 463, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 451, "optime" : { "t" : 1358375419000, "i" : 7813 }, "optimeDate" : ISODate("2013-01-16T22:30:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 450, "optime" : { "t" : 1358375415000, "i" : 7924 }, "optimeDate" : ISODate("2013-01-16T22:30:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 450, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:19Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 462, "optime" : { "t" : 1358375419000, "i" : 8211 }, "optimeDate" : ISODate("2013-01-16T22:30:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:21.320 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:22.053 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:30:22.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:30:23.320 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 468, "optime" : { "t" : 1358375424000, "i" : 1418 }, "optimeDate" : ISODate("2013-01-16T22:30:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 457, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 457, "optime" : { "t" : 1358375423000, "i" : 1831 }, "optimeDate" : ISODate("2013-01-16T22:30:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 467, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 455, "optime" : { "t" : 1358375423000, "i" : 7709 }, "optimeDate" : ISODate("2013-01-16T22:30:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 454, "optime" : { "t" : 1358375420000, "i" : 593 }, "optimeDate" : ISODate("2013-01-16T22:30:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 454, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:23Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 466, "optime" : { "t" : 1358375424000, "i" : 1388 }, "optimeDate" : ISODate("2013-01-16T22:30:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:25.320 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:27.054 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:27.321 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 472, "optime" : { "t" : 1358375428000, "i" : 3365 }, "optimeDate" : ISODate("2013-01-16T22:30:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 461, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 461, "optime" : { "t" : 1358375428000, "i" : 2169 }, "optimeDate" : ISODate("2013-01-16T22:30:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 471, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 459, "optime" : { "t" : 1358375427000, "i" : 5464 }, "optimeDate" : ISODate("2013-01-16T22:30:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 458, "optime" : { "t" : 1358375424000, "i" : 1418 }, "optimeDate" : ISODate("2013-01-16T22:30:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 458, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:27Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 470, "optime" : { "t" : 1358375428000, "i" : 3033 }, "optimeDate" : ISODate("2013-01-16T22:30:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:29.321 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:30:31.321 [conn29] end connection 127.0.0.2:46598 (2 connections now open) m31001| Wed Jan 16 17:30:31.321 [initandlisten] connection accepted from 127.0.0.2:56397 #31 (3 connections now open) m31000| Wed Jan 16 17:30:31.322 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:32.054 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 476, "optime" : { "t" : 1358375432000, "i" : 5530 }, "optimeDate" : ISODate("2013-01-16T22:30:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 465, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 465, "optime" : { "t" : 1358375428000, "i" : 3365 }, "optimeDate" : ISODate("2013-01-16T22:30:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 475, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 463, "optime" : { "t" : 1358375431000, "i" : 4026 }, "optimeDate" : ISODate("2013-01-16T22:30:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 462, "optime" : { "t" : 1358375432000, "i" : 5344 }, "optimeDate" : ISODate("2013-01-16T22:30:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 462, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 474, "optime" : { "t" : 1358375432000, "i" : 5530 }, "optimeDate" : ISODate("2013-01-16T22:30:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:30:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:30:33.322 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:30:34.571 [conn26] end connection 127.0.0.4:50921 (4 connections now open) m31000| Wed Jan 16 17:30:34.571 [initandlisten] connection accepted from 127.0.0.4:48163 #27 (5 connections now open) m31000| Wed Jan 16 17:30:35.323 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 480, "optime" : { "t" : 1358375436000, "i" : 6731 }, "optimeDate" : ISODate("2013-01-16T22:30:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 469, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 469, "optime" : { "t" : 1358375435000, "i" : 699 }, "optimeDate" : ISODate("2013-01-16T22:30:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 479, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 467, "optime" : { "t" : 1358375435000, "i" : 2300 }, "optimeDate" : ISODate("2013-01-16T22:30:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 466, "optime" : { "t" : 1358375436000, "i" : 4902 }, "optimeDate" : ISODate("2013-01-16T22:30:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 466, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:35Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 478, "optime" : { "t" : 1358375436000, "i" : 6346 }, "optimeDate" : ISODate("2013-01-16T22:30:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:30:37.055 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:37.323 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:30:39.323 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 484, "optime" : { "t" : 1358375440000, "i" : 8438 }, "optimeDate" : ISODate("2013-01-16T22:30:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 473, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 473, "optime" : { "t" : 1358375440000, "i" : 7435 }, "optimeDate" : ISODate("2013-01-16T22:30:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 483, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 471, "optime" : { "t" : 1358375439000, "i" : 1300 }, "optimeDate" : ISODate("2013-01-16T22:30:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 470, "optime" : { "t" : 1358375440000, "i" : 5487 }, "optimeDate" : ISODate("2013-01-16T22:30:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 470, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:39Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 482, "optime" : { "t" : 1358375440000, "i" : 8438 }, "optimeDate" : ISODate("2013-01-16T22:30:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:41.324 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:42.055 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:43.324 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 489, "optime" : { "t" : 1358375445000, "i" : 213 }, "optimeDate" : ISODate("2013-01-16T22:30:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 478, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 478, "optime" : { "t" : 1358375440000, "i" : 8438 }, "optimeDate" : ISODate("2013-01-16T22:30:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 488, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 476, "optime" : { "t" : 1358375443000, "i" : 53 }, "optimeDate" : ISODate("2013-01-16T22:30:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 475, "optime" : { "t" : 1358375444000, "i" : 4768 }, "optimeDate" : ISODate("2013-01-16T22:30:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 475, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 487, "optime" : { "t" : 1358375445000, "i" : 213 }, "optimeDate" : ISODate("2013-01-16T22:30:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:45.324 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:47.056 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:30:47.325 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:47.939 [conn32] end connection 127.0.0.3:34309 (3 connections now open) m31002| Wed Jan 16 17:30:47.939 [initandlisten] connection accepted from 127.0.0.3:53069 #33 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 493, "optime" : { "t" : 1358375449000, "i" : 1599 }, "optimeDate" : ISODate("2013-01-16T22:30:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 482, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 482, "optime" : { "t" : 1358375448000, "i" : 1844 }, "optimeDate" : ISODate("2013-01-16T22:30:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 492, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 480, "optime" : { "t" : 1358375445000, "i" : 213 }, "optimeDate" : ISODate("2013-01-16T22:30:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 479, "optime" : { "t" : 1358375448000, "i" : 4528 }, "optimeDate" : ISODate("2013-01-16T22:30:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 479, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:47Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 491, "optime" : { "t" : 1358375449000, "i" : 1491 }, "optimeDate" : ISODate("2013-01-16T22:30:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:49.325 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:30:51.326 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:30:52.057 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:30:53.303 [conn31] end connection 127.0.0.2:39672 (3 connections now open) m31000| Wed Jan 16 17:30:53.326 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 497, "optime" : { "t" : 1358375453000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:30:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 486, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 486, "optime" : { "t" : 1358375449000, "i" : 1599 }, "optimeDate" : ISODate("2013-01-16T22:30:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 496, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 484, "optime" : { "t" : 1358375449000, "i" : 1599 }, "optimeDate" : ISODate("2013-01-16T22:30:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 483, "optime" : { "t" : 1358375452000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-16T22:30:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 483, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 495, "optime" : { "t" : 1358375453000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:30:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:30:53.803 [initandlisten] connection accepted from 127.0.0.2:57217 #34 (4 connections now open) m31001| Wed Jan 16 17:30:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:30:55.326 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:30:57.057 [conn30] end connection 127.0.0.4:57185 (2 connections now open) m31000| Wed Jan 16 17:30:57.327 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31001| Wed Jan 16 17:30:57.557 [initandlisten] connection accepted from 127.0.0.4:46538 #32 (3 connections now open) m31002| Wed Jan 16 17:30:57.558 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 501, "optime" : { "t" : 1358375457000, "i" : 4423 }, "optimeDate" : ISODate("2013-01-16T22:30:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 490, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 490, "optime" : { "t" : 1358375453000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:30:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 500, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 488, "optime" : { "t" : 1358375453000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:30:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:30:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:30:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 487, "optime" : { "t" : 1358375456000, "i" : 1899 }, "optimeDate" : ISODate("2013-01-16T22:30:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 487, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 499, "optime" : { "t" : 1358375457000, "i" : 4423 }, "optimeDate" : ISODate("2013-01-16T22:30:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:30:59.327 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:31:01.327 [conn31] end connection 127.0.0.2:56397 (2 connections now open) m31001| Wed Jan 16 17:31:01.327 [initandlisten] connection accepted from 127.0.0.2:57901 #33 (3 connections now open) m31000| Wed Jan 16 17:31:01.327 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 505, "optime" : { "t" : 1358375461000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:31:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 494, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 494, "optime" : { "t" : 1358375461000, "i" : 2316 }, "optimeDate" : ISODate("2013-01-16T22:31:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 504, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 492, "optime" : { "t" : 1358375457000, "i" : 4423 }, "optimeDate" : ISODate("2013-01-16T22:30:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 491, "optime" : { "t" : 1358375457000, "i" : 4423 }, "optimeDate" : ISODate("2013-01-16T22:30:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 491, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 503, "optime" : { "t" : 1358375461000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:31:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:31:02.559 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:03.328 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:04.576 [conn27] end connection 127.0.0.4:48163 (4 connections now open) m31000| Wed Jan 16 17:31:04.576 [initandlisten] connection accepted from 127.0.0.4:57296 #28 (5 connections now open) m31001| Wed Jan 16 17:31:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:31:05.328 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 509, "optime" : { "t" : 1358375465000, "i" : 8246 }, "optimeDate" : ISODate("2013-01-16T22:31:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 498, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 498, "optime" : { "t" : 1358375461000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:31:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 508, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 496, "optime" : { "t" : 1358375465000, "i" : 8197 }, "optimeDate" : ISODate("2013-01-16T22:31:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 495, "optime" : { "t" : 1358375461000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:31:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 495, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:05Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 507, "optime" : { "t" : 1358375465000, "i" : 8197 }, "optimeDate" : ISODate("2013-01-16T22:31:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:07.328 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:07.559 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:09.329 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 514, "optime" : { "t" : 1358375470000, "i" : 1390 }, "optimeDate" : ISODate("2013-01-16T22:31:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 503, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 503, "optime" : { "t" : 1358375465000, "i" : 8246 }, "optimeDate" : ISODate("2013-01-16T22:31:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 513, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 501, "optime" : { "t" : 1358375469000, "i" : 8343 }, "optimeDate" : ISODate("2013-01-16T22:31:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 500, "optime" : { "t" : 1358375465000, "i" : 8246 }, "optimeDate" : ISODate("2013-01-16T22:31:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 500, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 512, "optime" : { "t" : 1358375470000, "i" : 1390 }, "optimeDate" : ISODate("2013-01-16T22:31:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:11.329 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:12.560 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:13.329 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 518, "optime" : { "t" : 1358375474000, "i" : 2348 }, "optimeDate" : ISODate("2013-01-16T22:31:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 507, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 507, "optime" : { "t" : 1358375473000, "i" : 6152 }, "optimeDate" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:30:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 517, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 505, "optime" : { "t" : 1358375473000, "i" : 7372 }, "optimeDate" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 504, "optime" : { "t" : 1358375470000, "i" : 1390 }, "optimeDate" : ISODate("2013-01-16T22:31:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 504, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:13Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 516, "optime" : { "t" : 1358375474000, "i" : 2348 }, "optimeDate" : ISODate("2013-01-16T22:31:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:15.330 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:17.330 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:17.561 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:31:17.945 [conn33] end connection 127.0.0.3:53069 (3 connections now open) m31002| Wed Jan 16 17:31:17.945 [initandlisten] connection accepted from 127.0.0.3:34153 #35 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 522, "optime" : { "t" : 1358375478000, "i" : 3653 }, "optimeDate" : ISODate("2013-01-16T22:31:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 511, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 511, "optime" : { "t" : 1358375474000, "i" : 2348 }, "optimeDate" : ISODate("2013-01-16T22:31:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 521, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 509, "optime" : { "t" : 1358375477000, "i" : 4899 }, "optimeDate" : ISODate("2013-01-16T22:31:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 508, "optime" : { "t" : 1358375474000, "i" : 2348 }, "optimeDate" : ISODate("2013-01-16T22:31:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 508, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:17Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 520, "optime" : { "t" : 1358375478000, "i" : 3443 }, "optimeDate" : ISODate("2013-01-16T22:31:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:19.331 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:21.331 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:22.561 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 526, "optime" : { "t" : 1358375482000, "i" : 4992 }, "optimeDate" : ISODate("2013-01-16T22:31:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 515, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 515, "optime" : { "t" : 1358375478000, "i" : 3653 }, "optimeDate" : ISODate("2013-01-16T22:31:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 525, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 513, "optime" : { "t" : 1358375481000, "i" : 4658 }, "optimeDate" : ISODate("2013-01-16T22:31:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 512, "optime" : { "t" : 1358375478000, "i" : 3653 }, "optimeDate" : ISODate("2013-01-16T22:31:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 512, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 524, "optime" : { "t" : 1358375482000, "i" : 4746 }, "optimeDate" : ISODate("2013-01-16T22:31:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:23.331 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:25.332 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 530, "optime" : { "t" : 1358375486000, "i" : 6506 }, "optimeDate" : ISODate("2013-01-16T22:31:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 519, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 519, "optime" : { "t" : 1358375486000, "i" : 2600 }, "optimeDate" : ISODate("2013-01-16T22:31:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 529, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 517, "optime" : { "t" : 1358375485000, "i" : 2860 }, "optimeDate" : ISODate("2013-01-16T22:31:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 516, "optime" : { "t" : 1358375486000, "i" : 5120 }, "optimeDate" : ISODate("2013-01-16T22:31:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 516, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:25Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 528, "optime" : { "t" : 1358375486000, "i" : 5532 }, "optimeDate" : ISODate("2013-01-16T22:31:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:31:26.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:31:27.332 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:27.562 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:29.333 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 534, "optime" : { "t" : 1358375490000, "i" : 7706 }, "optimeDate" : ISODate("2013-01-16T22:31:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 523, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 523, "optime" : { "t" : 1358375486000, "i" : 6506 }, "optimeDate" : ISODate("2013-01-16T22:31:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 533, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 521, "optime" : { "t" : 1358375489000, "i" : 1789 }, "optimeDate" : ISODate("2013-01-16T22:31:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 520, "optime" : { "t" : 1358375490000, "i" : 4927 }, "optimeDate" : ISODate("2013-01-16T22:31:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 520, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 532, "optime" : { "t" : 1358375490000, "i" : 7433 }, "optimeDate" : ISODate("2013-01-16T22:31:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:31:31.308 [conn34] end connection 127.0.0.2:57217 (3 connections now open) m31001| Wed Jan 16 17:31:31.333 [conn33] end connection 127.0.0.2:57901 (2 connections now open) m31001| Wed Jan 16 17:31:31.333 [initandlisten] connection accepted from 127.0.0.2:53682 #34 (3 connections now open) m31000| Wed Jan 16 17:31:31.333 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:31.808 [initandlisten] connection accepted from 127.0.0.2:59898 #36 (4 connections now open) m31002| Wed Jan 16 17:31:32.563 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:33.334 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:34.166 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.3, filling with zeroes... m31000| Wed Jan 16 17:31:34.581 [conn28] end connection 127.0.0.4:57296 (4 connections now open) m31000| Wed Jan 16 17:31:34.581 [initandlisten] connection accepted from 127.0.0.4:40384 #29 (5 connections now open) STATUS FROM : 127.0.0.2:31000 m31001| Wed Jan 16 17:31:35.063 [conn32] end connection 127.0.0.4:46538 (2 connections now open) m31000| Wed Jan 16 17:31:35.190 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.3, size: 128MB, took 1.024 secs m31000| Wed Jan 16 17:31:35.191 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:1025378 1025ms m31002| Wed Jan 16 17:31:35.192 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.3, filling with zeroes... m31000| Wed Jan 16 17:31:35.334 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:31:35.563 [initandlisten] connection accepted from 127.0.0.4:56473 #35 (3 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 539, "optime" : { "t" : 1358375495000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-16T22:31:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 528, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 528, "optime" : { "t" : 1358375494000, "i" : 1730 }, "optimeDate" : ISODate("2013-01-16T22:31:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 538, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 526, "optime" : { "t" : 1358375493000, "i" : 296 }, "optimeDate" : ISODate("2013-01-16T22:31:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:35Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 525, "optime" : { "t" : 1358375494000, "i" : 1730 }, "optimeDate" : ISODate("2013-01-16T22:31:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 525, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 537, "optime" : { "t" : 1358375494000, "i" : 1730 }, "optimeDate" : ISODate("2013-01-16T22:31:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:31:36.233 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.3, size: 128MB, took 1.04 secs m31001| Wed Jan 16 17:31:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:31:37.334 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:38.064 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:39.335 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:39.499 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:1742498 871ms STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:31:39.750 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:362626 181ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 544, "optime" : { "t" : 1358375500000, "i" : 3732 }, "optimeDate" : ISODate("2013-01-16T22:31:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 533, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 533, "optime" : { "t" : 1358375498000, "i" : 51 }, "optimeDate" : ISODate("2013-01-16T22:31:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 543, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 531, "optime" : { "t" : 1358375499000, "i" : 5364 }, "optimeDate" : ISODate("2013-01-16T22:31:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 530, "optime" : { "t" : 1358375495000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-16T22:31:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 530, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:39Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 542, "optime" : { "t" : 1358375500000, "i" : 3126 }, "optimeDate" : ISODate("2013-01-16T22:31:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:41.335 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:43.064 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:43.335 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 548, "optime" : { "t" : 1358375504000, "i" : 2622 }, "optimeDate" : ISODate("2013-01-16T22:31:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 537, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 537, "optime" : { "t" : 1358375500000, "i" : 3732 }, "optimeDate" : ISODate("2013-01-16T22:31:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 547, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 535, "optime" : { "t" : 1358375503000, "i" : 6457 }, "optimeDate" : ISODate("2013-01-16T22:31:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 534, "optime" : { "t" : 1358375500000, "i" : 3732 }, "optimeDate" : ISODate("2013-01-16T22:31:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 534, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:43Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 546, "optime" : { "t" : 1358375504000, "i" : 2164 }, "optimeDate" : ISODate("2013-01-16T22:31:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:45.336 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:47.336 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:47.949 [conn35] end connection 127.0.0.3:34153 (3 connections now open) m31002| Wed Jan 16 17:31:47.950 [initandlisten] connection accepted from 127.0.0.3:48867 #37 (4 connections now open) m31002| Wed Jan 16 17:31:48.065 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 552, "optime" : { "t" : 1358375508000, "i" : 5167 }, "optimeDate" : ISODate("2013-01-16T22:31:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 541, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 541, "optime" : { "t" : 1358375504000, "i" : 2622 }, "optimeDate" : ISODate("2013-01-16T22:31:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 551, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 539, "optime" : { "t" : 1358375507000, "i" : 4833 }, "optimeDate" : ISODate("2013-01-16T22:31:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 538, "optime" : { "t" : 1358375504000, "i" : 2622 }, "optimeDate" : ISODate("2013-01-16T22:31:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 538, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 550, "optime" : { "t" : 1358375508000, "i" : 5115 }, "optimeDate" : ISODate("2013-01-16T22:31:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:49.336 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:51.337 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 556, "optime" : { "t" : 1358375512000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:31:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 545, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 545, "optime" : { "t" : 1358375511000, "i" : 4162 }, "optimeDate" : ISODate("2013-01-16T22:31:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 555, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 543, "optime" : { "t" : 1358375511000, "i" : 5462 }, "optimeDate" : ISODate("2013-01-16T22:31:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 542, "optime" : { "t" : 1358375508000, "i" : 5167 }, "optimeDate" : ISODate("2013-01-16T22:31:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 542, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:51Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 554, "optime" : { "t" : 1358375512000, "i" : 3835 }, "optimeDate" : ISODate("2013-01-16T22:31:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:31:53.066 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:31:53.337 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:31:55.337 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 560, "optime" : { "t" : 1358375516000, "i" : 6045 }, "optimeDate" : ISODate("2013-01-16T22:31:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 549, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 549, "optime" : { "t" : 1358375512000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:31:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 559, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 547, "optime" : { "t" : 1358375515000, "i" : 3715 }, "optimeDate" : ISODate("2013-01-16T22:31:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:31:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 546, "optime" : { "t" : 1358375516000, "i" : 5508 }, "optimeDate" : ISODate("2013-01-16T22:31:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 546, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:55Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 558, "optime" : { "t" : 1358375516000, "i" : 5954 }, "optimeDate" : ISODate("2013-01-16T22:31:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:31:57.338 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:31:58.066 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:31:58.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:31:59.338 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 564, "optime" : { "t" : 1358375520000, "i" : 7684 }, "optimeDate" : ISODate("2013-01-16T22:32:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 553, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 553, "optime" : { "t" : 1358375516000, "i" : 6045 }, "optimeDate" : ISODate("2013-01-16T22:31:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:31:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 551, "optime" : { "t" : 1358375519000, "i" : 1826 }, "optimeDate" : ISODate("2013-01-16T22:31:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 550, "optime" : { "t" : 1358375520000, "i" : 5369 }, "optimeDate" : ISODate("2013-01-16T22:32:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 550, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 562, "optime" : { "t" : 1358375520000, "i" : 7684 }, "optimeDate" : ISODate("2013-01-16T22:32:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:32:01.338 [conn34] end connection 127.0.0.2:53682 (2 connections now open) m31001| Wed Jan 16 17:32:01.339 [initandlisten] connection accepted from 127.0.0.2:51169 #36 (3 connections now open) m31000| Wed Jan 16 17:32:01.339 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:03.067 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:03.339 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:04.587 [conn29] end connection 127.0.0.4:40384 (4 connections now open) m31000| Wed Jan 16 17:32:04.587 [initandlisten] connection accepted from 127.0.0.4:50253 #30 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 568, "optime" : { "t" : 1358375524000, "i" : 8363 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 557, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 557, "optime" : { "t" : 1358375524000, "i" : 2512 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 567, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 555, "optime" : { "t" : 1358375523000, "i" : 498 }, "optimeDate" : ISODate("2013-01-16T22:32:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 554, "optime" : { "t" : 1358375524000, "i" : 4868 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 554, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:03Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 566, "optime" : { "t" : 1358375524000, "i" : 8019 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:05.340 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:07.340 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:08.068 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:32:08.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 573, "optime" : { "t" : 1358375529000, "i" : 782 }, "optimeDate" : ISODate("2013-01-16T22:32:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 562, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 562, "optime" : { "t" : 1358375524000, "i" : 8363 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 572, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 560, "optime" : { "t" : 1358375524000, "i" : 8363 }, "optimeDate" : ISODate("2013-01-16T22:32:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 559, "optime" : { "t" : 1358375528000, "i" : 4978 }, "optimeDate" : ISODate("2013-01-16T22:32:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 559, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 571, "optime" : { "t" : 1358375529000, "i" : 782 }, "optimeDate" : ISODate("2013-01-16T22:32:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:32:09.313 [conn36] end connection 127.0.0.2:59898 (3 connections now open) m31000| Wed Jan 16 17:32:09.340 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:09.813 [initandlisten] connection accepted from 127.0.0.2:51451 #38 (4 connections now open) m31000| Wed Jan 16 17:32:11.341 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:32:13.068 [conn35] end connection 127.0.0.4:56473 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 577, "optime" : { "t" : 1358375533000, "i" : 2875 }, "optimeDate" : ISODate("2013-01-16T22:32:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 566, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 566, "optime" : { "t" : 1358375532000, "i" : 1089 }, "optimeDate" : ISODate("2013-01-16T22:32:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 576, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 564, "optime" : { "t" : 1358375529000, "i" : 782 }, "optimeDate" : ISODate("2013-01-16T22:32:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 563, "optime" : { "t" : 1358375532000, "i" : 3549 }, "optimeDate" : ISODate("2013-01-16T22:32:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:11Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 575, "optime" : { "t" : 1358375533000, "i" : 2178 }, "optimeDate" : ISODate("2013-01-16T22:32:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:13.341 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:32:13.568 [initandlisten] connection accepted from 127.0.0.4:38774 #37 (3 connections now open) m31002| Wed Jan 16 17:32:13.569 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:15.342 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:17.342 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 581, "optime" : { "t" : 1358375537000, "i" : 4310 }, "optimeDate" : ISODate("2013-01-16T22:32:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 570, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 570, "optime" : { "t" : 1358375537000, "i" : 2845 }, "optimeDate" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:31:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 580, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 568, "optime" : { "t" : 1358375533000, "i" : 2875 }, "optimeDate" : ISODate("2013-01-16T22:32:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 567, "optime" : { "t" : 1358375536000, "i" : 1938 }, "optimeDate" : ISODate("2013-01-16T22:32:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 567, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 579, "optime" : { "t" : 1358375537000, "i" : 4027 }, "optimeDate" : ISODate("2013-01-16T22:32:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:32:17.955 [conn37] end connection 127.0.0.3:48867 (3 connections now open) m31002| Wed Jan 16 17:32:17.955 [initandlisten] connection accepted from 127.0.0.3:55416 #39 (4 connections now open) m31002| Wed Jan 16 17:32:18.569 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:19.342 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:21.343 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 585, "optime" : { "t" : 1358375541000, "i" : 5746 }, "optimeDate" : ISODate("2013-01-16T22:32:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 574, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 574, "optime" : { "t" : 1358375537000, "i" : 4310 }, "optimeDate" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 584, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 572, "optime" : { "t" : 1358375537000, "i" : 4310 }, "optimeDate" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:21Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 571, "optime" : { "t" : 1358375540000, "i" : 295 }, "optimeDate" : ISODate("2013-01-16T22:32:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 571, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 583, "optime" : { "t" : 1358375541000, "i" : 5023 }, "optimeDate" : ISODate("2013-01-16T22:32:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:23.343 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:23.570 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:25.343 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 589, "optime" : { "t" : 1358375545000, "i" : 7361 }, "optimeDate" : ISODate("2013-01-16T22:32:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 578, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 578, "optime" : { "t" : 1358375544000, "i" : 727 }, "optimeDate" : ISODate("2013-01-16T22:32:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 588, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 576, "optime" : { "t" : 1358375541000, "i" : 5746 }, "optimeDate" : ISODate("2013-01-16T22:32:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 575, "optime" : { "t" : 1358375541000, "i" : 5746 }, "optimeDate" : ISODate("2013-01-16T22:32:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 575, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:23Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 587, "optime" : { "t" : 1358375545000, "i" : 7227 }, "optimeDate" : ISODate("2013-01-16T22:32:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:27.344 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:28.571 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:29.344 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 594, "optime" : { "t" : 1358375550000, "i" : 385 }, "optimeDate" : ISODate("2013-01-16T22:32:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 583, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 583, "optime" : { "t" : 1358375549000, "i" : 7270 }, "optimeDate" : ISODate("2013-01-16T22:32:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 593, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 581, "optime" : { "t" : 1358375549000, "i" : 8464 }, "optimeDate" : ISODate("2013-01-16T22:32:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 580, "optime" : { "t" : 1358375545000, "i" : 7361 }, "optimeDate" : ISODate("2013-01-16T22:32:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 580, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:29Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 592, "optime" : { "t" : 1358375550000, "i" : 385 }, "optimeDate" : ISODate("2013-01-16T22:32:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:32:30.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:32:31.344 [conn36] end connection 127.0.0.2:51169 (2 connections now open) m31001| Wed Jan 16 17:32:31.344 [initandlisten] connection accepted from 127.0.0.2:51296 #38 (3 connections now open) m31000| Wed Jan 16 17:32:31.345 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:33.345 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:33.571 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 598, "optime" : { "t" : 1358375554000, "i" : 1460 }, "optimeDate" : ISODate("2013-01-16T22:32:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 587, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 587, "optime" : { "t" : 1358375550000, "i" : 385 }, "optimeDate" : ISODate("2013-01-16T22:32:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 597, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 585, "optime" : { "t" : 1358375553000, "i" : 8137 }, "optimeDate" : ISODate("2013-01-16T22:32:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 584, "optime" : { "t" : 1358375550000, "i" : 385 }, "optimeDate" : ISODate("2013-01-16T22:32:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 584, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:33Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 596, "optime" : { "t" : 1358375554000, "i" : 1354 }, "optimeDate" : ISODate("2013-01-16T22:32:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:34.592 [conn30] end connection 127.0.0.4:50253 (4 connections now open) m31000| Wed Jan 16 17:32:34.592 [initandlisten] connection accepted from 127.0.0.4:41182 #31 (5 connections now open) m31000| Wed Jan 16 17:32:35.345 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:37.346 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:38.232 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:365281 182ms STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:32:38.458 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:312792 156ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 602, "optime" : { "t" : 1358375558000, "i" : 2866 }, "optimeDate" : ISODate("2013-01-16T22:32:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 591, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 591, "optime" : { "t" : 1358375557000, "i" : 1079 }, "optimeDate" : ISODate("2013-01-16T22:32:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 601, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 589, "optime" : { "t" : 1358375557000, "i" : 6561 }, "optimeDate" : ISODate("2013-01-16T22:32:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 588, "optime" : { "t" : 1358375554000, "i" : 1460 }, "optimeDate" : ISODate("2013-01-16T22:32:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 588, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:37Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 600, "optime" : { "t" : 1358375558000, "i" : 2601 }, "optimeDate" : ISODate("2013-01-16T22:32:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:32:38.572 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:39.346 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:32:40.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:32:41.346 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 606, "optime" : { "t" : 1358375562000, "i" : 5497 }, "optimeDate" : ISODate("2013-01-16T22:32:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 595, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 595, "optime" : { "t" : 1358375562000, "i" : 2731 }, "optimeDate" : ISODate("2013-01-16T22:32:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 605, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 593, "optime" : { "t" : 1358375561000, "i" : 4340 }, "optimeDate" : ISODate("2013-01-16T22:32:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 592, "optime" : { "t" : 1358375562000, "i" : 5388 }, "optimeDate" : ISODate("2013-01-16T22:32:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 592, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:41Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 604, "optime" : { "t" : 1358375562000, "i" : 5476 }, "optimeDate" : ISODate("2013-01-16T22:32:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:43.347 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:43.573 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:45.347 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 610, "optime" : { "t" : 1358375566000, "i" : 6551 }, "optimeDate" : ISODate("2013-01-16T22:32:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 599, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 599, "optime" : { "t" : 1358375562000, "i" : 5497 }, "optimeDate" : ISODate("2013-01-16T22:32:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 609, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 597, "optime" : { "t" : 1358375565000, "i" : 2756 }, "optimeDate" : ISODate("2013-01-16T22:32:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 596, "optime" : { "t" : 1358375566000, "i" : 5073 }, "optimeDate" : ISODate("2013-01-16T22:32:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 596, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 608, "optime" : { "t" : 1358375566000, "i" : 6547 }, "optimeDate" : ISODate("2013-01-16T22:32:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:32:47.318 [conn38] end connection 127.0.0.2:51451 (3 connections now open) m31000| Wed Jan 16 17:32:47.348 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:47.818 [initandlisten] connection accepted from 127.0.0.2:50446 #40 (4 connections now open) m31002| Wed Jan 16 17:32:47.960 [conn39] end connection 127.0.0.3:55416 (3 connections now open) m31002| Wed Jan 16 17:32:47.960 [initandlisten] connection accepted from 127.0.0.3:42919 #41 (4 connections now open) m31002| Wed Jan 16 17:32:48.573 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:32:49.348 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 614, "optime" : { "t" : 1358375570000, "i" : 8047 }, "optimeDate" : ISODate("2013-01-16T22:32:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 603, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 603, "optime" : { "t" : 1358375570000, "i" : 2430 }, "optimeDate" : ISODate("2013-01-16T22:32:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 613, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 601, "optime" : { "t" : 1358375569000, "i" : 1687 }, "optimeDate" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 600, "optime" : { "t" : 1358375570000, "i" : 4912 }, "optimeDate" : ISODate("2013-01-16T22:32:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 600, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:49Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 612, "optime" : { "t" : 1358375570000, "i" : 8047 }, "optimeDate" : ISODate("2013-01-16T22:32:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:32:51.073 [conn37] end connection 127.0.0.4:38774 (2 connections now open) m31000| Wed Jan 16 17:32:51.348 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:32:51.574 [initandlisten] connection accepted from 127.0.0.4:40157 #39 (3 connections now open) m31000| Wed Jan 16 17:32:53.349 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:54.074 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 619, "optime" : { "t" : 1358375575000, "i" : 253 }, "optimeDate" : ISODate("2013-01-16T22:32:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 608, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 608, "optime" : { "t" : 1358375570000, "i" : 8047 }, "optimeDate" : ISODate("2013-01-16T22:32:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 618, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 606, "optime" : { "t" : 1358375573000, "i" : 86 }, "optimeDate" : ISODate("2013-01-16T22:32:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 605, "optime" : { "t" : 1358375574000, "i" : 5434 }, "optimeDate" : ISODate("2013-01-16T22:32:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 605, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 617, "optime" : { "t" : 1358375575000, "i" : 253 }, "optimeDate" : ISODate("2013-01-16T22:32:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:55.349 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:32:57.350 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:32:59.075 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 623, "optime" : { "t" : 1358375579000, "i" : 2095 }, "optimeDate" : ISODate("2013-01-16T22:32:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 612, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 612, "optime" : { "t" : 1358375575000, "i" : 253 }, "optimeDate" : ISODate("2013-01-16T22:32:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 622, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 610, "optime" : { "t" : 1358375575000, "i" : 253 }, "optimeDate" : ISODate("2013-01-16T22:32:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:32:59Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:32:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 609, "optime" : { "t" : 1358375578000, "i" : 4845 }, "optimeDate" : ISODate("2013-01-16T22:32:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 609, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 621, "optime" : { "t" : 1358375579000, "i" : 2095 }, "optimeDate" : ISODate("2013-01-16T22:32:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:32:59.350 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:33:01.350 [conn38] end connection 127.0.0.2:51296 (2 connections now open) m31001| Wed Jan 16 17:33:01.351 [initandlisten] connection accepted from 127.0.0.2:46129 #40 (3 connections now open) m31000| Wed Jan 16 17:33:01.351 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:33:02.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 627, "optime" : { "t" : 1358375583000, "i" : 2672 }, "optimeDate" : ISODate("2013-01-16T22:33:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 616, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 616, "optime" : { "t" : 1358375582000, "i" : 5411 }, "optimeDate" : ISODate("2013-01-16T22:33:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 626, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 614, "optime" : { "t" : 1358375579000, "i" : 2095 }, "optimeDate" : ISODate("2013-01-16T22:32:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 613, "optime" : { "t" : 1358375582000, "i" : 3594 }, "optimeDate" : ISODate("2013-01-16T22:33:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 613, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:01Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 625, "optime" : { "t" : 1358375583000, "i" : 1898 }, "optimeDate" : ISODate("2013-01-16T22:33:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:03.351 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:04.075 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:04.597 [conn31] end connection 127.0.0.4:41182 (4 connections now open) m31000| Wed Jan 16 17:33:04.597 [initandlisten] connection accepted from 127.0.0.4:40588 #32 (5 connections now open) m31000| Wed Jan 16 17:33:05.352 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:07.352 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 631, "optime" : { "t" : 1358375587000, "i" : 4274 }, "optimeDate" : ISODate("2013-01-16T22:33:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 620, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 620, "optime" : { "t" : 1358375583000, "i" : 2672 }, "optimeDate" : ISODate("2013-01-16T22:33:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 630, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 618, "optime" : { "t" : 1358375583000, "i" : 2672 }, "optimeDate" : ISODate("2013-01-16T22:33:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 617, "optime" : { "t" : 1358375586000, "i" : 2365 }, "optimeDate" : ISODate("2013-01-16T22:33:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 617, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 629, "optime" : { "t" : 1358375587000, "i" : 4190 }, "optimeDate" : ISODate("2013-01-16T22:33:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:33:09.076 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:09.352 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:11.353 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 635, "optime" : { "t" : 1358375591000, "i" : 5538 }, "optimeDate" : ISODate("2013-01-16T22:33:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 624, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 624, "optime" : { "t" : 1358375587000, "i" : 4274 }, "optimeDate" : ISODate("2013-01-16T22:33:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 634, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 622, "optime" : { "t" : 1358375587000, "i" : 4274 }, "optimeDate" : ISODate("2013-01-16T22:33:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 621, "optime" : { "t" : 1358375590000, "i" : 730 }, "optimeDate" : ISODate("2013-01-16T22:33:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 621, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 633, "optime" : { "t" : 1358375591000, "i" : 5538 }, "optimeDate" : ISODate("2013-01-16T22:33:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:33:12.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:33:13.353 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:14.077 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:15.353 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 639, "optime" : { "t" : 1358375595000, "i" : 7139 }, "optimeDate" : ISODate("2013-01-16T22:33:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 628, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 628, "optime" : { "t" : 1358375595000, "i" : 2665 }, "optimeDate" : ISODate("2013-01-16T22:33:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 638, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 626, "optime" : { "t" : 1358375591000, "i" : 5538 }, "optimeDate" : ISODate("2013-01-16T22:33:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 625, "optime" : { "t" : 1358375591000, "i" : 5538 }, "optimeDate" : ISODate("2013-01-16T22:33:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 625, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 637, "optime" : { "t" : 1358375595000, "i" : 7042 }, "optimeDate" : ISODate("2013-01-16T22:33:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:17.354 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:17.966 [conn41] end connection 127.0.0.3:42919 (3 connections now open) m31002| Wed Jan 16 17:33:17.966 [initandlisten] connection accepted from 127.0.0.3:48877 #42 (4 connections now open) m31002| Wed Jan 16 17:33:19.077 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:19.354 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 643, "optime" : { "t" : 1358375599000, "i" : 8669 }, "optimeDate" : ISODate("2013-01-16T22:33:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 632, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 632, "optime" : { "t" : 1358375595000, "i" : 7139 }, "optimeDate" : ISODate("2013-01-16T22:33:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:32:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 642, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 630, "optime" : { "t" : 1358375599000, "i" : 8585 }, "optimeDate" : ISODate("2013-01-16T22:33:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 629, "optime" : { "t" : 1358375595000, "i" : 7139 }, "optimeDate" : ISODate("2013-01-16T22:33:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 629, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:19Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 641, "optime" : { "t" : 1358375599000, "i" : 8669 }, "optimeDate" : ISODate("2013-01-16T22:33:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:21.354 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:23.355 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:24.078 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 648, "optime" : { "t" : 1358375604000, "i" : 1566 }, "optimeDate" : ISODate("2013-01-16T22:33:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 637, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 637, "optime" : { "t" : 1358375599000, "i" : 8669 }, "optimeDate" : ISODate("2013-01-16T22:33:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 647, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 635, "optime" : { "t" : 1358375603000, "i" : 8252 }, "optimeDate" : ISODate("2013-01-16T22:33:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:24Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 634, "optime" : { "t" : 1358375599000, "i" : 8669 }, "optimeDate" : ISODate("2013-01-16T22:33:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:22Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 634, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 646, "optime" : { "t" : 1358375604000, "i" : 1566 }, "optimeDate" : ISODate("2013-01-16T22:33:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:33:25.324 [conn40] end connection 127.0.0.2:50446 (3 connections now open) m31000| Wed Jan 16 17:33:25.355 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:25.824 [initandlisten] connection accepted from 127.0.0.2:59635 #43 (4 connections now open) m31000| Wed Jan 16 17:33:27.355 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 652, "optime" : { "t" : 1358375608000, "i" : 2760 }, "optimeDate" : ISODate("2013-01-16T22:33:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 641, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 641, "optime" : { "t" : 1358375604000, "i" : 1566 }, "optimeDate" : ISODate("2013-01-16T22:33:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 651, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 639, "optime" : { "t" : 1358375607000, "i" : 7111 }, "optimeDate" : ISODate("2013-01-16T22:33:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 638, "optime" : { "t" : 1358375604000, "i" : 1566 }, "optimeDate" : ISODate("2013-01-16T22:33:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 638, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:27Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 650, "optime" : { "t" : 1358375608000, "i" : 2735 }, "optimeDate" : ISODate("2013-01-16T22:33:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:33:29.079 [conn39] end connection 127.0.0.4:40157 (2 connections now open) m31000| Wed Jan 16 17:33:29.356 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:33:29.579 [initandlisten] connection accepted from 127.0.0.4:42132 #41 (3 connections now open) m31002| Wed Jan 16 17:33:29.579 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:33:31.356 [conn40] end connection 127.0.0.2:46129 (2 connections now open) m31001| Wed Jan 16 17:33:31.356 [initandlisten] connection accepted from 127.0.0.2:37200 #42 (3 connections now open) m31000| Wed Jan 16 17:33:31.356 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 656, "optime" : { "t" : 1358375612000, "i" : 4265 }, "optimeDate" : ISODate("2013-01-16T22:33:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 645, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 645, "optime" : { "t" : 1358375608000, "i" : 2760 }, "optimeDate" : ISODate("2013-01-16T22:33:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 655, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 643, "optime" : { "t" : 1358375611000, "i" : 5500 }, "optimeDate" : ISODate("2013-01-16T22:33:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 642, "optime" : { "t" : 1358375608000, "i" : 2760 }, "optimeDate" : ISODate("2013-01-16T22:33:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 642, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 654, "optime" : { "t" : 1358375612000, "i" : 4265 }, "optimeDate" : ISODate("2013-01-16T22:33:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:33.357 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:34.580 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:34.603 [conn32] end connection 127.0.0.4:40588 (4 connections now open) m31000| Wed Jan 16 17:33:34.603 [initandlisten] connection accepted from 127.0.0.4:56564 #33 (5 connections now open) m31001| Wed Jan 16 17:33:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:33:35.357 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 660, "optime" : { "t" : 1358375616000, "i" : 6580 }, "optimeDate" : ISODate("2013-01-16T22:33:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 649, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 649, "optime" : { "t" : 1358375615000, "i" : 2069 }, "optimeDate" : ISODate("2013-01-16T22:33:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 659, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 647, "optime" : { "t" : 1358375615000, "i" : 3419 }, "optimeDate" : ISODate("2013-01-16T22:33:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 646, "optime" : { "t" : 1358375616000, "i" : 5679 }, "optimeDate" : ISODate("2013-01-16T22:33:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 646, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:35Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 658, "optime" : { "t" : 1358375616000, "i" : 6151 }, "optimeDate" : ISODate("2013-01-16T22:33:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:37.357 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:39.358 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:39.580 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 664, "optime" : { "t" : 1358375620000, "i" : 7289 }, "optimeDate" : ISODate("2013-01-16T22:33:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 653, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 653, "optime" : { "t" : 1358375616000, "i" : 6580 }, "optimeDate" : ISODate("2013-01-16T22:33:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 663, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 651, "optime" : { "t" : 1358375619000, "i" : 1988 }, "optimeDate" : ISODate("2013-01-16T22:33:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 650, "optime" : { "t" : 1358375620000, "i" : 5818 }, "optimeDate" : ISODate("2013-01-16T22:33:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 650, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:39Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 662, "optime" : { "t" : 1358375620000, "i" : 7289 }, "optimeDate" : ISODate("2013-01-16T22:33:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:41.358 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:43.359 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:44.581 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:33:44.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 668, "optime" : { "t" : 1358375624000, "i" : 7989 }, "optimeDate" : ISODate("2013-01-16T22:33:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 657, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 657, "optime" : { "t" : 1358375620000, "i" : 7289 }, "optimeDate" : ISODate("2013-01-16T22:33:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 667, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 655, "optime" : { "t" : 1358375623000, "i" : 1755 }, "optimeDate" : ISODate("2013-01-16T22:33:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 654, "optime" : { "t" : 1358375624000, "i" : 5511 }, "optimeDate" : ISODate("2013-01-16T22:33:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 654, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 666, "optime" : { "t" : 1358375624000, "i" : 7952 }, "optimeDate" : ISODate("2013-01-16T22:33:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:45.359 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:47.360 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:47.971 [conn42] end connection 127.0.0.3:48877 (3 connections now open) m31002| Wed Jan 16 17:33:47.971 [initandlisten] connection accepted from 127.0.0.3:55812 #44 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 673, "optime" : { "t" : 1358375629000, "i" : 497 }, "optimeDate" : ISODate("2013-01-16T22:33:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 662, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 662, "optime" : { "t" : 1358375628000, "i" : 2678 }, "optimeDate" : ISODate("2013-01-16T22:33:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 672, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 660, "optime" : { "t" : 1358375627000, "i" : 318 }, "optimeDate" : ISODate("2013-01-16T22:33:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 659, "optime" : { "t" : 1358375628000, "i" : 4647 }, "optimeDate" : ISODate("2013-01-16T22:33:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 659, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:47Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 671, "optime" : { "t" : 1358375629000, "i" : 497 }, "optimeDate" : ISODate("2013-01-16T22:33:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:49.360 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:49.582 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:33:51.360 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:33:53.017 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:310249 155ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 677, "optime" : { "t" : 1358375633000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:33:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 666, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 666, "optime" : { "t" : 1358375629000, "i" : 497 }, "optimeDate" : ISODate("2013-01-16T22:33:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 676, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 664, "optime" : { "t" : 1358375629000, "i" : 497 }, "optimeDate" : ISODate("2013-01-16T22:33:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 663, "optime" : { "t" : 1358375632000, "i" : 4321 }, "optimeDate" : ISODate("2013-01-16T22:33:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 663, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 675, "optime" : { "t" : 1358375633000, "i" : 2621 }, "optimeDate" : ISODate("2013-01-16T22:33:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:53.361 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:54.582 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:33:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:33:55.361 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:33:57.361 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 681, "optime" : { "t" : 1358375637000, "i" : 2922 }, "optimeDate" : ISODate("2013-01-16T22:33:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 670, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 670, "optime" : { "t" : 1358375633000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:33:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 680, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 668, "optime" : { "t" : 1358375633000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:33:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:33:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 667, "optime" : { "t" : 1358375636000, "i" : 3239 }, "optimeDate" : ISODate("2013-01-16T22:33:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 667, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 679, "optime" : { "t" : 1358375637000, "i" : 2922 }, "optimeDate" : ISODate("2013-01-16T22:33:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:33:59.362 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:33:59.583 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:34:01.362 [conn42] end connection 127.0.0.2:37200 (2 connections now open) m31001| Wed Jan 16 17:34:01.362 [initandlisten] connection accepted from 127.0.0.2:48099 #43 (3 connections now open) m31000| Wed Jan 16 17:34:01.362 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 685, "optime" : { "t" : 1358375641000, "i" : 4517 }, "optimeDate" : ISODate("2013-01-16T22:34:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 674, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 674, "optime" : { "t" : 1358375640000, "i" : 3817 }, "optimeDate" : ISODate("2013-01-16T22:34:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 684, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 672, "optime" : { "t" : 1358375637000, "i" : 2922 }, "optimeDate" : ISODate("2013-01-16T22:33:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 671, "optime" : { "t" : 1358375640000, "i" : 1668 }, "optimeDate" : ISODate("2013-01-16T22:34:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 671, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:33:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:33:59Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 683, "optime" : { "t" : 1358375641000, "i" : 4169 }, "optimeDate" : ISODate("2013-01-16T22:34:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:34:03.329 [conn43] end connection 127.0.0.2:59635 (3 connections now open) m31000| Wed Jan 16 17:34:03.363 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:03.829 [initandlisten] connection accepted from 127.0.0.2:52902 #45 (4 connections now open) m31002| Wed Jan 16 17:34:04.583 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:04.608 [conn33] end connection 127.0.0.4:56564 (4 connections now open) m31000| Wed Jan 16 17:34:04.608 [initandlisten] connection accepted from 127.0.0.4:55988 #34 (5 connections now open) m31000| Wed Jan 16 17:34:05.363 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 689, "optime" : { "t" : 1358375645000, "i" : 5494 }, "optimeDate" : ISODate("2013-01-16T22:34:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 678, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 678, "optime" : { "t" : 1358375641000, "i" : 4517 }, "optimeDate" : ISODate("2013-01-16T22:34:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 688, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 676, "optime" : { "t" : 1358375641000, "i" : 4517 }, "optimeDate" : ISODate("2013-01-16T22:34:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 675, "optime" : { "t" : 1358375644000, "i" : 919 }, "optimeDate" : ISODate("2013-01-16T22:34:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 675, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 687, "optime" : { "t" : 1358375645000, "i" : 5494 }, "optimeDate" : ISODate("2013-01-16T22:34:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:34:07.084 [conn41] end connection 127.0.0.4:42132 (2 connections now open) m31000| Wed Jan 16 17:34:07.363 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:34:07.584 [initandlisten] connection accepted from 127.0.0.4:43283 #44 (3 connections now open) m31000| Wed Jan 16 17:34:09.364 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 693, "optime" : { "t" : 1358375649000, "i" : 6986 }, "optimeDate" : ISODate("2013-01-16T22:34:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 682, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 682, "optime" : { "t" : 1358375648000, "i" : 1398 }, "optimeDate" : ISODate("2013-01-16T22:34:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 692, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 680, "optime" : { "t" : 1358375645000, "i" : 5494 }, "optimeDate" : ISODate("2013-01-16T22:34:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 679, "optime" : { "t" : 1358375645000, "i" : 5494 }, "optimeDate" : ISODate("2013-01-16T22:34:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 679, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:07Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 691, "optime" : { "t" : 1358375649000, "i" : 6986 }, "optimeDate" : ISODate("2013-01-16T22:34:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:34:10.084 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:11.364 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:13.364 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 697, "optime" : { "t" : 1358375653000, "i" : 8931 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 686, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 686, "optime" : { "t" : 1358375653000, "i" : 7131 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 696, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 684, "optime" : { "t" : 1358375653000, "i" : 8477 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 683, "optime" : { "t" : 1358375649000, "i" : 6986 }, "optimeDate" : ISODate("2013-01-16T22:34:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 683, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:13Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 695, "optime" : { "t" : 1358375653000, "i" : 8572 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:34:15.085 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:15.365 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:34:16.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:34:17.365 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:17.952 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293687 146ms m31002| Wed Jan 16 17:34:17.977 [conn44] end connection 127.0.0.3:55812 (3 connections now open) m31002| Wed Jan 16 17:34:17.977 [initandlisten] connection accepted from 127.0.0.3:35820 #46 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 702, "optime" : { "t" : 1358375658000, "i" : 1605 }, "optimeDate" : ISODate("2013-01-16T22:34:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 691, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 691, "optime" : { "t" : 1358375653000, "i" : 8931 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 701, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 689, "optime" : { "t" : 1358375657000, "i" : 7798 }, "optimeDate" : ISODate("2013-01-16T22:34:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 688, "optime" : { "t" : 1358375653000, "i" : 8931 }, "optimeDate" : ISODate("2013-01-16T22:34:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 688, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:17Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 700, "optime" : { "t" : 1358375658000, "i" : 378 }, "optimeDate" : ISODate("2013-01-16T22:34:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:19.365 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:20.086 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:21.366 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 706, "optime" : { "t" : 1358375662000, "i" : 2326 }, "optimeDate" : ISODate("2013-01-16T22:34:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 695, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 695, "optime" : { "t" : 1358375661000, "i" : 1579 }, "optimeDate" : ISODate("2013-01-16T22:34:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 705, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 693, "optime" : { "t" : 1358375661000, "i" : 7217 }, "optimeDate" : ISODate("2013-01-16T22:34:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 692, "optime" : { "t" : 1358375658000, "i" : 1605 }, "optimeDate" : ISODate("2013-01-16T22:34:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 692, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:21Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 704, "optime" : { "t" : 1358375662000, "i" : 2174 }, "optimeDate" : ISODate("2013-01-16T22:34:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:23.366 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:25.086 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:25.366 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 710, "optime" : { "t" : 1358375666000, "i" : 4397 }, "optimeDate" : ISODate("2013-01-16T22:34:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 699, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 699, "optime" : { "t" : 1358375666000, "i" : 3202 }, "optimeDate" : ISODate("2013-01-16T22:34:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 709, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 697, "optime" : { "t" : 1358375665000, "i" : 5602 }, "optimeDate" : ISODate("2013-01-16T22:34:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 696, "optime" : { "t" : 1358375662000, "i" : 2326 }, "optimeDate" : ISODate("2013-01-16T22:34:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 696, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:25Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 708, "optime" : { "t" : 1358375666000, "i" : 4351 }, "optimeDate" : ISODate("2013-01-16T22:34:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:34:26.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:34:27.367 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:29.367 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:30.087 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 714, "optime" : { "t" : 1358375670000, "i" : 5452 }, "optimeDate" : ISODate("2013-01-16T22:34:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 703, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 703, "optime" : { "t" : 1358375666000, "i" : 4397 }, "optimeDate" : ISODate("2013-01-16T22:34:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 713, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 701, "optime" : { "t" : 1358375669000, "i" : 4229 }, "optimeDate" : ISODate("2013-01-16T22:34:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 700, "optime" : { "t" : 1358375670000, "i" : 4750 }, "optimeDate" : ISODate("2013-01-16T22:34:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 700, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 712, "optime" : { "t" : 1358375670000, "i" : 5446 }, "optimeDate" : ISODate("2013-01-16T22:34:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:34:31.367 [conn43] end connection 127.0.0.2:48099 (2 connections now open) m31001| Wed Jan 16 17:34:31.368 [initandlisten] connection accepted from 127.0.0.2:32791 #45 (3 connections now open) m31000| Wed Jan 16 17:34:31.368 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:33.368 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:34.613 [conn34] end connection 127.0.0.4:55988 (4 connections now open) m31000| Wed Jan 16 17:34:34.613 [initandlisten] connection accepted from 127.0.0.4:50276 #35 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 718, "optime" : { "t" : 1358375674000, "i" : 6950 }, "optimeDate" : ISODate("2013-01-16T22:34:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 707, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 707, "optime" : { "t" : 1358375673000, "i" : 1555 }, "optimeDate" : ISODate("2013-01-16T22:34:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 717, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 705, "optime" : { "t" : 1358375673000, "i" : 3042 }, "optimeDate" : ISODate("2013-01-16T22:34:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 704, "optime" : { "t" : 1358375674000, "i" : 5523 }, "optimeDate" : ISODate("2013-01-16T22:34:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 704, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:33Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 716, "optime" : { "t" : 1358375674000, "i" : 6950 }, "optimeDate" : ISODate("2013-01-16T22:34:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:34:35.088 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:35.369 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:37.369 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:38.136 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:244307 122ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 722, "optime" : { "t" : 1358375678000, "i" : 8276 }, "optimeDate" : ISODate("2013-01-16T22:34:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 711, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 711, "optime" : { "t" : 1358375678000, "i" : 7737 }, "optimeDate" : ISODate("2013-01-16T22:34:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 721, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 709, "optime" : { "t" : 1358375677000, "i" : 1541 }, "optimeDate" : ISODate("2013-01-16T22:34:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 708, "optime" : { "t" : 1358375678000, "i" : 6119 }, "optimeDate" : ISODate("2013-01-16T22:34:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 708, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:37Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 720, "optime" : { "t" : 1358375678000, "i" : 7737 }, "optimeDate" : ISODate("2013-01-16T22:34:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:39.369 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:40.088 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:34:41.334 [conn45] end connection 127.0.0.2:52902 (3 connections now open) m31000| Wed Jan 16 17:34:41.370 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:41.834 [initandlisten] connection accepted from 127.0.0.2:43484 #47 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 727, "optime" : { "t" : 1358375683000, "i" : 48 }, "optimeDate" : ISODate("2013-01-16T22:34:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 716, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 716, "optime" : { "t" : 1358375678000, "i" : 8276 }, "optimeDate" : ISODate("2013-01-16T22:34:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 726, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 714, "optime" : { "t" : 1358375681000, "i" : 931 }, "optimeDate" : ISODate("2013-01-16T22:34:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 713, "optime" : { "t" : 1358375682000, "i" : 5326 }, "optimeDate" : ISODate("2013-01-16T22:34:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 713, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 725, "optime" : { "t" : 1358375683000, "i" : 48 }, "optimeDate" : ISODate("2013-01-16T22:34:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:43.370 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:34:45.089 [conn44] end connection 127.0.0.4:43283 (2 connections now open) m31000| Wed Jan 16 17:34:45.371 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:34:45.589 [initandlisten] connection accepted from 127.0.0.4:40629 #46 (3 connections now open) m31002| Wed Jan 16 17:34:45.589 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 731, "optime" : { "t" : 1358375687000, "i" : 1525 }, "optimeDate" : ISODate("2013-01-16T22:34:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 720, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 720, "optime" : { "t" : 1358375686000, "i" : 6706 }, "optimeDate" : ISODate("2013-01-16T22:34:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 730, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 718, "optime" : { "t" : 1358375683000, "i" : 48 }, "optimeDate" : ISODate("2013-01-16T22:34:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 717, "optime" : { "t" : 1358375686000, "i" : 5091 }, "optimeDate" : ISODate("2013-01-16T22:34:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 717, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:45Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 729, "optime" : { "t" : 1358375687000, "i" : 1525 }, "optimeDate" : ISODate("2013-01-16T22:34:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:47.371 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:47.981 [conn46] end connection 127.0.0.3:35820 (3 connections now open) m31002| Wed Jan 16 17:34:47.982 [initandlisten] connection accepted from 127.0.0.3:53463 #48 (4 connections now open) m31001| Wed Jan 16 17:34:48.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:34:49.371 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:34:50.590 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 735, "optime" : { "t" : 1358375691000, "i" : 2619 }, "optimeDate" : ISODate("2013-01-16T22:34:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 724, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 724, "optime" : { "t" : 1358375687000, "i" : 1525 }, "optimeDate" : ISODate("2013-01-16T22:34:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 734, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 722, "optime" : { "t" : 1358375687000, "i" : 1525 }, "optimeDate" : ISODate("2013-01-16T22:34:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 721, "optime" : { "t" : 1358375690000, "i" : 3553 }, "optimeDate" : ISODate("2013-01-16T22:34:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 721, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 733, "optime" : { "t" : 1358375691000, "i" : 2619 }, "optimeDate" : ISODate("2013-01-16T22:34:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:34:51.372 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:53.372 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:34:55.372 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 739, "optime" : { "t" : 1358375695000, "i" : 4176 }, "optimeDate" : ISODate("2013-01-16T22:34:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 728, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 728, "optime" : { "t" : 1358375694000, "i" : 68 }, "optimeDate" : ISODate("2013-01-16T22:34:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 738, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 726, "optime" : { "t" : 1358375691000, "i" : 2619 }, "optimeDate" : ISODate("2013-01-16T22:34:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 725, "optime" : { "t" : 1358375694000, "i" : 2237 }, "optimeDate" : ISODate("2013-01-16T22:34:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 725, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:53Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 737, "optime" : { "t" : 1358375695000, "i" : 4148 }, "optimeDate" : ISODate("2013-01-16T22:34:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:34:55.590 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:34:57.373 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:34:58.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:34:59.373 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 743, "optime" : { "t" : 1358375699000, "i" : 5770 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 732, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 732, "optime" : { "t" : 1358375699000, "i" : 2815 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 742, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 730, "optime" : { "t" : 1358375695000, "i" : 4176 }, "optimeDate" : ISODate("2013-01-16T22:34:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:34:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 729, "optime" : { "t" : 1358375698000, "i" : 502 }, "optimeDate" : ISODate("2013-01-16T22:34:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:34:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 729, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 741, "optime" : { "t" : 1358375699000, "i" : 5344 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:35:00.591 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:35:01.373 [conn45] end connection 127.0.0.2:32791 (2 connections now open) m31001| Wed Jan 16 17:35:01.374 [initandlisten] connection accepted from 127.0.0.2:40680 #47 (3 connections now open) m31000| Wed Jan 16 17:35:01.374 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:35:03.129 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:643789 321ms m31000| Wed Jan 16 17:35:03.374 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 747, "optime" : { "t" : 1358375703000, "i" : 8375 }, "optimeDate" : ISODate("2013-01-16T22:35:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 736, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 736, "optime" : { "t" : 1358375699000, "i" : 5770 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 746, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 734, "optime" : { "t" : 1358375699000, "i" : 5770 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:03Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 733, "optime" : { "t" : 1358375699000, "i" : 5770 }, "optimeDate" : ISODate("2013-01-16T22:34:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 733, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 745, "optime" : { "t" : 1358375703000, "i" : 8375 }, "optimeDate" : ISODate("2013-01-16T22:35:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:04.618 [conn35] end connection 127.0.0.4:50276 (4 connections now open) m31000| Wed Jan 16 17:35:04.618 [initandlisten] connection accepted from 127.0.0.4:36266 #36 (5 connections now open) m31000| Wed Jan 16 17:35:05.374 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:05.592 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:07.375 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 751, "optime" : { "t" : 1358375707000, "i" : 6798 }, "optimeDate" : ISODate("2013-01-16T22:35:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 740, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 740, "optime" : { "t" : 1358375706000, "i" : 1622 }, "optimeDate" : ISODate("2013-01-16T22:35:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:34:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 750, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 738, "optime" : { "t" : 1358375703000, "i" : 8375 }, "optimeDate" : ISODate("2013-01-16T22:35:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 737, "optime" : { "t" : 1358375703000, "i" : 8375 }, "optimeDate" : ISODate("2013-01-16T22:35:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 737, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:05Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 749, "optime" : { "t" : 1358375707000, "i" : 6798 }, "optimeDate" : ISODate("2013-01-16T22:35:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:09.375 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:10.592 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:11.376 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 755, "optime" : { "t" : 1358375711000, "i" : 8104 }, "optimeDate" : ISODate("2013-01-16T22:35:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 744, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 744, "optime" : { "t" : 1358375711000, "i" : 7033 }, "optimeDate" : ISODate("2013-01-16T22:35:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 754, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 742, "optime" : { "t" : 1358375707000, "i" : 6798 }, "optimeDate" : ISODate("2013-01-16T22:35:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 741, "optime" : { "t" : 1358375707000, "i" : 6798 }, "optimeDate" : ISODate("2013-01-16T22:35:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 741, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 753, "optime" : { "t" : 1358375711000, "i" : 8104 }, "optimeDate" : ISODate("2013-01-16T22:35:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:13.376 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:35:15.376 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:15.593 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 760, "optime" : { "t" : 1358375716000, "i" : 315 }, "optimeDate" : ISODate("2013-01-16T22:35:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 749, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 749, "optime" : { "t" : 1358375711000, "i" : 8104 }, "optimeDate" : ISODate("2013-01-16T22:35:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 759, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 747, "optime" : { "t" : 1358375715000, "i" : 8633 }, "optimeDate" : ISODate("2013-01-16T22:35:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 746, "optime" : { "t" : 1358375711000, "i" : 8104 }, "optimeDate" : ISODate("2013-01-16T22:35:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 746, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:15Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 758, "optime" : { "t" : 1358375716000, "i" : 315 }, "optimeDate" : ISODate("2013-01-16T22:35:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:17.377 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:17.987 [conn48] end connection 127.0.0.3:53463 (3 connections now open) m31002| Wed Jan 16 17:35:17.987 [initandlisten] connection accepted from 127.0.0.3:36433 #49 (4 connections now open) m31002| Wed Jan 16 17:35:19.339 [conn47] end connection 127.0.0.2:43484 (3 connections now open) m31000| Wed Jan 16 17:35:19.377 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:19.839 [initandlisten] connection accepted from 127.0.0.2:50301 #50 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 764, "optime" : { "t" : 1358375720000, "i" : 1997 }, "optimeDate" : ISODate("2013-01-16T22:35:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 753, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 753, "optime" : { "t" : 1358375719000, "i" : 6335 }, "optimeDate" : ISODate("2013-01-16T22:35:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 763, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 751, "optime" : { "t" : 1358375719000, "i" : 7457 }, "optimeDate" : ISODate("2013-01-16T22:35:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 750, "optime" : { "t" : 1358375716000, "i" : 315 }, "optimeDate" : ISODate("2013-01-16T22:35:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 750, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:19Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 762, "optime" : { "t" : 1358375720000, "i" : 1997 }, "optimeDate" : ISODate("2013-01-16T22:35:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:35:20.594 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:35:20.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:35:21.377 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:35:23.094 [conn46] end connection 127.0.0.4:40629 (2 connections now open) m31000| Wed Jan 16 17:35:23.378 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:35:23.594 [initandlisten] connection accepted from 127.0.0.4:49200 #48 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 768, "optime" : { "t" : 1358375724000, "i" : 3285 }, "optimeDate" : ISODate("2013-01-16T22:35:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 757, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 757, "optime" : { "t" : 1358375720000, "i" : 1997 }, "optimeDate" : ISODate("2013-01-16T22:35:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 767, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 755, "optime" : { "t" : 1358375723000, "i" : 5896 }, "optimeDate" : ISODate("2013-01-16T22:35:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 754, "optime" : { "t" : 1358375720000, "i" : 1997 }, "optimeDate" : ISODate("2013-01-16T22:35:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 754, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:23Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 766, "optime" : { "t" : 1358375724000, "i" : 3098 }, "optimeDate" : ISODate("2013-01-16T22:35:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:25.378 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:26.095 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:27.378 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 772, "optime" : { "t" : 1358375728000, "i" : 5803 }, "optimeDate" : ISODate("2013-01-16T22:35:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 761, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 761, "optime" : { "t" : 1358375724000, "i" : 3285 }, "optimeDate" : ISODate("2013-01-16T22:35:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 771, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 759, "optime" : { "t" : 1358375727000, "i" : 3624 }, "optimeDate" : ISODate("2013-01-16T22:35:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 758, "optime" : { "t" : 1358375724000, "i" : 3285 }, "optimeDate" : ISODate("2013-01-16T22:35:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 758, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:27Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 770, "optime" : { "t" : 1358375728000, "i" : 5803 }, "optimeDate" : ISODate("2013-01-16T22:35:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:29.379 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:35:30.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:35:31.095 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:35:31.379 [conn47] end connection 127.0.0.2:40680 (2 connections now open) m31001| Wed Jan 16 17:35:31.379 [initandlisten] connection accepted from 127.0.0.2:51251 #49 (3 connections now open) m31000| Wed Jan 16 17:35:31.379 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 776, "optime" : { "t" : 1358375732000, "i" : 6069 }, "optimeDate" : ISODate("2013-01-16T22:35:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 765, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 765, "optime" : { "t" : 1358375732000, "i" : 3216 }, "optimeDate" : ISODate("2013-01-16T22:35:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 775, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 763, "optime" : { "t" : 1358375731000, "i" : 3922 }, "optimeDate" : ISODate("2013-01-16T22:35:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 762, "optime" : { "t" : 1358375732000, "i" : 5328 }, "optimeDate" : ISODate("2013-01-16T22:35:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 762, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:31Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 774, "optime" : { "t" : 1358375732000, "i" : 6069 }, "optimeDate" : ISODate("2013-01-16T22:35:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:33.380 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:35:34.624 [conn36] end connection 127.0.0.4:36266 (4 connections now open) m31000| Wed Jan 16 17:35:34.624 [initandlisten] connection accepted from 127.0.0.4:50140 #37 (5 connections now open) m31000| Wed Jan 16 17:35:35.380 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:36.096 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 780, "optime" : { "t" : 1358375736000, "i" : 7031 }, "optimeDate" : ISODate("2013-01-16T22:35:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 769, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 769, "optime" : { "t" : 1358375732000, "i" : 6069 }, "optimeDate" : ISODate("2013-01-16T22:35:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 779, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 767, "optime" : { "t" : 1358375735000, "i" : 2865 }, "optimeDate" : ISODate("2013-01-16T22:35:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 766, "optime" : { "t" : 1358375736000, "i" : 4974 }, "optimeDate" : ISODate("2013-01-16T22:35:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 766, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 778, "optime" : { "t" : 1358375736000, "i" : 7031 }, "optimeDate" : ISODate("2013-01-16T22:35:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:37.381 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:35:39.381 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 785, "optime" : { "t" : 1358375741000, "i" : 257 }, "optimeDate" : ISODate("2013-01-16T22:35:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 774, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 774, "optime" : { "t" : 1358375736000, "i" : 7031 }, "optimeDate" : ISODate("2013-01-16T22:35:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 784, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 772, "optime" : { "t" : 1358375739000, "i" : 1195 }, "optimeDate" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 771, "optime" : { "t" : 1358375740000, "i" : 5184 }, "optimeDate" : ISODate("2013-01-16T22:35:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 771, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:39Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 783, "optime" : { "t" : 1358375741000, "i" : 29 }, "optimeDate" : ISODate("2013-01-16T22:35:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:35:41.096 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:41.382 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:35:43.382 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 789, "optime" : { "t" : 1358375745000, "i" : 1505 }, "optimeDate" : ISODate("2013-01-16T22:35:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 778, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 778, "optime" : { "t" : 1358375744000, "i" : 6767 }, "optimeDate" : ISODate("2013-01-16T22:35:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 788, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 776, "optime" : { "t" : 1358375741000, "i" : 257 }, "optimeDate" : ISODate("2013-01-16T22:35:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 775, "optime" : { "t" : 1358375744000, "i" : 5258 }, "optimeDate" : ISODate("2013-01-16T22:35:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:44Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 775, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:43Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 787, "optime" : { "t" : 1358375745000, "i" : 1505 }, "optimeDate" : ISODate("2013-01-16T22:35:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:45.382 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:46.097 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:47.383 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:47.992 [conn49] end connection 127.0.0.3:36433 (3 connections now open) m31002| Wed Jan 16 17:35:47.993 [initandlisten] connection accepted from 127.0.0.3:42355 #51 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 793, "optime" : { "t" : 1358375749000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:35:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 782, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 782, "optime" : { "t" : 1358375745000, "i" : 1505 }, "optimeDate" : ISODate("2013-01-16T22:35:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 792, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 780, "optime" : { "t" : 1358375745000, "i" : 1505 }, "optimeDate" : ISODate("2013-01-16T22:35:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 779, "optime" : { "t" : 1358375748000, "i" : 3902 }, "optimeDate" : ISODate("2013-01-16T22:35:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 779, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 791, "optime" : { "t" : 1358375749000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:35:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:49.383 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:51.098 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:35:51.384 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:35:52.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:35:53.384 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 797, "optime" : { "t" : 1358375753000, "i" : 3595 }, "optimeDate" : ISODate("2013-01-16T22:35:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 786, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 786, "optime" : { "t" : 1358375752000, "i" : 83 }, "optimeDate" : ISODate("2013-01-16T22:35:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 796, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 784, "optime" : { "t" : 1358375749000, "i" : 3050 }, "optimeDate" : ISODate("2013-01-16T22:35:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 783, "optime" : { "t" : 1358375752000, "i" : 2936 }, "optimeDate" : ISODate("2013-01-16T22:35:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 783, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:51Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 795, "optime" : { "t" : 1358375753000, "i" : 2795 }, "optimeDate" : ISODate("2013-01-16T22:35:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:35:55.384 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:35:56.099 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:35:57.345 [conn50] end connection 127.0.0.2:50301 (3 connections now open) m31000| Wed Jan 16 17:35:57.385 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 801, "optime" : { "t" : 1358375757000, "i" : 5352 }, "optimeDate" : ISODate("2013-01-16T22:35:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 790, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 790, "optime" : { "t" : 1358375753000, "i" : 3595 }, "optimeDate" : ISODate("2013-01-16T22:35:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 800, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 788, "optime" : { "t" : 1358375753000, "i" : 3595 }, "optimeDate" : ISODate("2013-01-16T22:35:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:35:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:35:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 787, "optime" : { "t" : 1358375756000, "i" : 1913 }, "optimeDate" : ISODate("2013-01-16T22:35:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 787, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 799, "optime" : { "t" : 1358375757000, "i" : 4848 }, "optimeDate" : ISODate("2013-01-16T22:35:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:35:57.845 [initandlisten] connection accepted from 127.0.0.2:41856 #52 (4 connections now open) m31000| Wed Jan 16 17:35:59.385 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:36:01.099 [conn48] end connection 127.0.0.4:49200 (2 connections now open) m31001| Wed Jan 16 17:36:01.385 [conn49] end connection 127.0.0.2:51251 (1 connection now open) m31001| Wed Jan 16 17:36:01.385 [initandlisten] connection accepted from 127.0.0.2:49426 #50 (2 connections now open) m31000| Wed Jan 16 17:36:01.385 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:36:01.599 [initandlisten] connection accepted from 127.0.0.4:52149 #51 (3 connections now open) m31002| Wed Jan 16 17:36:01.599 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 805, "optime" : { "t" : 1358375761000, "i" : 6040 }, "optimeDate" : ISODate("2013-01-16T22:36:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 794, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 794, "optime" : { "t" : 1358375757000, "i" : 5352 }, "optimeDate" : ISODate("2013-01-16T22:35:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 804, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 792, "optime" : { "t" : 1358375757000, "i" : 5352 }, "optimeDate" : ISODate("2013-01-16T22:35:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:01Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 791, "optime" : { "t" : 1358375760000, "i" : 750 }, "optimeDate" : ISODate("2013-01-16T22:36:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 791, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 803, "optime" : { "t" : 1358375761000, "i" : 6040 }, "optimeDate" : ISODate("2013-01-16T22:36:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:36:02.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:36:03.386 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:04.629 [conn37] end connection 127.0.0.4:50140 (4 connections now open) m31000| Wed Jan 16 17:36:04.629 [initandlisten] connection accepted from 127.0.0.4:51541 #38 (5 connections now open) m31000| Wed Jan 16 17:36:05.386 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 809, "optime" : { "t" : 1358375765000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 798, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 798, "optime" : { "t" : 1358375765000, "i" : 2625 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 808, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 796, "optime" : { "t" : 1358375761000, "i" : 6040 }, "optimeDate" : ISODate("2013-01-16T22:36:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 795, "optime" : { "t" : 1358375761000, "i" : 6040 }, "optimeDate" : ISODate("2013-01-16T22:36:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 795, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 807, "optime" : { "t" : 1358375765000, "i" : 7124 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:36:06.600 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:07.386 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:09.387 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 813, "optime" : { "t" : 1358375769000, "i" : 7892 }, "optimeDate" : ISODate("2013-01-16T22:36:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 802, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 802, "optime" : { "t" : 1358375765000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:35:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 812, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 800, "optime" : { "t" : 1358375765000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 799, "optime" : { "t" : 1358375765000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:36:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 799, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 811, "optime" : { "t" : 1358375769000, "i" : 7346 }, "optimeDate" : ISODate("2013-01-16T22:36:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:11.387 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:11.601 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:13.274 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:746038 373ms m31000| Wed Jan 16 17:36:13.387 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 817, "optime" : { "t" : 1358375773000, "i" : 9769 }, "optimeDate" : ISODate("2013-01-16T22:36:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 806, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 806, "optime" : { "t" : 1358375769000, "i" : 7892 }, "optimeDate" : ISODate("2013-01-16T22:36:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 816, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 804, "optime" : { "t" : 1358375769000, "i" : 7892 }, "optimeDate" : ISODate("2013-01-16T22:36:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 803, "optime" : { "t" : 1358375769000, "i" : 7892 }, "optimeDate" : ISODate("2013-01-16T22:36:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:12Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 803, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:11Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 815, "optime" : { "t" : 1358375773000, "i" : 9769 }, "optimeDate" : ISODate("2013-01-16T22:36:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:15.388 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:16.601 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:17.388 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 821, "optime" : { "t" : 1358375777000, "i" : 8458 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 810, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 810, "optime" : { "t" : 1358375777000, "i" : 8412 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 820, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 808, "optime" : { "t" : 1358375773000, "i" : 9769 }, "optimeDate" : ISODate("2013-01-16T22:36:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 807, "optime" : { "t" : 1358375773000, "i" : 9769 }, "optimeDate" : ISODate("2013-01-16T22:36:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 807, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 819, "optime" : { "t" : 1358375777000, "i" : 8458 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:36:17.998 [conn51] end connection 127.0.0.3:42355 (3 connections now open) m31002| Wed Jan 16 17:36:17.999 [initandlisten] connection accepted from 127.0.0.3:33934 #53 (5 connections now open) m31000| Wed Jan 16 17:36:19.388 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:21.389 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:21.602 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 825, "optime" : { "t" : 1358375781000, "i" : 8906 }, "optimeDate" : ISODate("2013-01-16T22:36:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 814, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 814, "optime" : { "t" : 1358375777000, "i" : 8458 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 824, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 812, "optime" : { "t" : 1358375777000, "i" : 8458 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:21Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 811, "optime" : { "t" : 1358375777000, "i" : 8458 }, "optimeDate" : ISODate("2013-01-16T22:36:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 811, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 823, "optime" : { "t" : 1358375781000, "i" : 8906 }, "optimeDate" : ISODate("2013-01-16T22:36:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:23.389 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:36:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:36:25.389 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 830, "optime" : { "t" : 1358375786000, "i" : 943 }, "optimeDate" : ISODate("2013-01-16T22:36:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 819, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 819, "optime" : { "t" : 1358375785000, "i" : 2704 }, "optimeDate" : ISODate("2013-01-16T22:36:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 829, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 817, "optime" : { "t" : 1358375785000, "i" : 8249 }, "optimeDate" : ISODate("2013-01-16T22:36:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 816, "optime" : { "t" : 1358375781000, "i" : 8906 }, "optimeDate" : ISODate("2013-01-16T22:36:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 816, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:25Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 828, "optime" : { "t" : 1358375786000, "i" : 456 }, "optimeDate" : ISODate("2013-01-16T22:36:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:36:26.603 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:27.390 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:29.390 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 834, "optime" : { "t" : 1358375790000, "i" : 2475 }, "optimeDate" : ISODate("2013-01-16T22:36:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 823, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 823, "optime" : { "t" : 1358375786000, "i" : 943 }, "optimeDate" : ISODate("2013-01-16T22:36:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 833, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 821, "optime" : { "t" : 1358375789000, "i" : 7343 }, "optimeDate" : ISODate("2013-01-16T22:36:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 820, "optime" : { "t" : 1358375786000, "i" : 943 }, "optimeDate" : ISODate("2013-01-16T22:36:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 820, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:29Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 832, "optime" : { "t" : 1358375790000, "i" : 2475 }, "optimeDate" : ISODate("2013-01-16T22:36:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:36:31.390 [conn50] end connection 127.0.0.2:49426 (2 connections now open) m31001| Wed Jan 16 17:36:31.390 [initandlisten] connection accepted from 127.0.0.2:45394 #52 (3 connections now open) m31000| Wed Jan 16 17:36:31.391 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:31.603 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:33.391 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 838, "optime" : { "t" : 1358375794000, "i" : 2858 }, "optimeDate" : ISODate("2013-01-16T22:36:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 827, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 827, "optime" : { "t" : 1358375790000, "i" : 2475 }, "optimeDate" : ISODate("2013-01-16T22:36:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 837, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 825, "optime" : { "t" : 1358375793000, "i" : 6604 }, "optimeDate" : ISODate("2013-01-16T22:36:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 824, "optime" : { "t" : 1358375790000, "i" : 2475 }, "optimeDate" : ISODate("2013-01-16T22:36:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 824, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 836, "optime" : { "t" : 1358375794000, "i" : 2374 }, "optimeDate" : ISODate("2013-01-16T22:36:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:34.635 [conn38] end connection 127.0.0.4:51541 (4 connections now open) m31000| Wed Jan 16 17:36:34.635 [initandlisten] connection accepted from 127.0.0.4:36990 #39 (5 connections now open) m31001| Wed Jan 16 17:36:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:36:35.350 [conn52] end connection 127.0.0.2:41856 (3 connections now open) m31000| Wed Jan 16 17:36:35.391 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:35.850 [initandlisten] connection accepted from 127.0.0.2:44591 #54 (4 connections now open) m31002| Wed Jan 16 17:36:36.604 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:37.392 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:38.259 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:244012 122ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 842, "optime" : { "t" : 1358375798000, "i" : 4339 }, "optimeDate" : ISODate("2013-01-16T22:36:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 831, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 831, "optime" : { "t" : 1358375798000, "i" : 2549 }, "optimeDate" : ISODate("2013-01-16T22:36:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 841, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 829, "optime" : { "t" : 1358375797000, "i" : 5660 }, "optimeDate" : ISODate("2013-01-16T22:36:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 828, "optime" : { "t" : 1358375794000, "i" : 2858 }, "optimeDate" : ISODate("2013-01-16T22:36:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 828, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:37Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 840, "optime" : { "t" : 1358375798000, "i" : 4175 }, "optimeDate" : ISODate("2013-01-16T22:36:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:36:39.104 [conn51] end connection 127.0.0.4:52149 (2 connections now open) m31000| Wed Jan 16 17:36:39.392 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:36:39.604 [initandlisten] connection accepted from 127.0.0.4:50938 #53 (3 connections now open) m31000| Wed Jan 16 17:36:41.392 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:42.105 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 846, "optime" : { "t" : 1358375802000, "i" : 5608 }, "optimeDate" : ISODate("2013-01-16T22:36:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 835, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 835, "optime" : { "t" : 1358375798000, "i" : 4339 }, "optimeDate" : ISODate("2013-01-16T22:36:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 845, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 833, "optime" : { "t" : 1358375801000, "i" : 4384 }, "optimeDate" : ISODate("2013-01-16T22:36:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 832, "optime" : { "t" : 1358375798000, "i" : 4339 }, "optimeDate" : ISODate("2013-01-16T22:36:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 832, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 844, "optime" : { "t" : 1358375802000, "i" : 5477 }, "optimeDate" : ISODate("2013-01-16T22:36:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:43.393 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:36:45.393 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 850, "optime" : { "t" : 1358375806000, "i" : 7046 }, "optimeDate" : ISODate("2013-01-16T22:36:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 839, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 839, "optime" : { "t" : 1358375805000, "i" : 1503 }, "optimeDate" : ISODate("2013-01-16T22:36:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 849, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 837, "optime" : { "t" : 1358375805000, "i" : 2948 }, "optimeDate" : ISODate("2013-01-16T22:36:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 836, "optime" : { "t" : 1358375806000, "i" : 5510 }, "optimeDate" : ISODate("2013-01-16T22:36:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 836, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 848, "optime" : { "t" : 1358375806000, "i" : 7029 }, "optimeDate" : ISODate("2013-01-16T22:36:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:36:47.106 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:47.394 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:48.003 [conn53] end connection 127.0.0.3:33934 (3 connections now open) m31002| Wed Jan 16 17:36:48.003 [initandlisten] connection accepted from 127.0.0.3:60111 #55 (4 connections now open) m31000| Wed Jan 16 17:36:49.394 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 854, "optime" : { "t" : 1358375810000, "i" : 8372 }, "optimeDate" : ISODate("2013-01-16T22:36:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 843, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 843, "optime" : { "t" : 1358375810000, "i" : 7533 }, "optimeDate" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 853, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 841, "optime" : { "t" : 1358375809000, "i" : 1623 }, "optimeDate" : ISODate("2013-01-16T22:36:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 840, "optime" : { "t" : 1358375810000, "i" : 5438 }, "optimeDate" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 840, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 852, "optime" : { "t" : 1358375810000, "i" : 7674 }, "optimeDate" : ISODate("2013-01-16T22:36:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:51.394 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:36:52.106 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:53.395 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 859, "optime" : { "t" : 1358375815000, "i" : 375 }, "optimeDate" : ISODate("2013-01-16T22:36:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 848, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 848, "optime" : { "t" : 1358375810000, "i" : 8372 }, "optimeDate" : ISODate("2013-01-16T22:36:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 858, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 846, "optime" : { "t" : 1358375813000, "i" : 36 }, "optimeDate" : ISODate("2013-01-16T22:36:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 845, "optime" : { "t" : 1358375814000, "i" : 5507 }, "optimeDate" : ISODate("2013-01-16T22:36:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 845, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 857, "optime" : { "t" : 1358375815000, "i" : 375 }, "optimeDate" : ISODate("2013-01-16T22:36:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:55.395 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:36:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:36:57.107 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:36:57.396 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 863, "optime" : { "t" : 1358375819000, "i" : 2066 }, "optimeDate" : ISODate("2013-01-16T22:36:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 852, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 852, "optime" : { "t" : 1358375818000, "i" : 2074 }, "optimeDate" : ISODate("2013-01-16T22:36:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 862, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 850, "optime" : { "t" : 1358375815000, "i" : 375 }, "optimeDate" : ISODate("2013-01-16T22:36:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:36:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 849, "optime" : { "t" : 1358375818000, "i" : 4988 }, "optimeDate" : ISODate("2013-01-16T22:36:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 849, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:36:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 861, "optime" : { "t" : 1358375819000, "i" : 2066 }, "optimeDate" : ISODate("2013-01-16T22:36:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:36:59.396 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:01.396 [conn52] end connection 127.0.0.2:45394 (2 connections now open) m31001| Wed Jan 16 17:37:01.396 [initandlisten] connection accepted from 127.0.0.2:43980 #54 (3 connections now open) m31000| Wed Jan 16 17:37:01.397 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:02.108 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:02.911 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:210556 105ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 867, "optime" : { "t" : 1358375823000, "i" : 2794 }, "optimeDate" : ISODate("2013-01-16T22:37:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 856, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 856, "optime" : { "t" : 1358375819000, "i" : 2066 }, "optimeDate" : ISODate("2013-01-16T22:36:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 866, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 854, "optime" : { "t" : 1358375819000, "i" : 2066 }, "optimeDate" : ISODate("2013-01-16T22:36:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 853, "optime" : { "t" : 1358375822000, "i" : 3984 }, "optimeDate" : ISODate("2013-01-16T22:37:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 853, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 865, "optime" : { "t" : 1358375823000, "i" : 2794 }, "optimeDate" : ISODate("2013-01-16T22:37:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:03.397 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:37:04.640 [conn39] end connection 127.0.0.4:36990 (4 connections now open) m31000| Wed Jan 16 17:37:04.640 [initandlisten] connection accepted from 127.0.0.4:33869 #40 (5 connections now open) m31000| Wed Jan 16 17:37:05.397 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:37:07.108 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:07.398 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 871, "optime" : { "t" : 1358375827000, "i" : 4296 }, "optimeDate" : ISODate("2013-01-16T22:37:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 860, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 860, "optime" : { "t" : 1358375823000, "i" : 2794 }, "optimeDate" : ISODate("2013-01-16T22:37:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 870, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 858, "optime" : { "t" : 1358375823000, "i" : 2794 }, "optimeDate" : ISODate("2013-01-16T22:37:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:07Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 857, "optime" : { "t" : 1358375826000, "i" : 2747 }, "optimeDate" : ISODate("2013-01-16T22:37:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 857, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 869, "optime" : { "t" : 1358375827000, "i" : 4143 }, "optimeDate" : ISODate("2013-01-16T22:37:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:09.398 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:37:11.398 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 875, "optime" : { "t" : 1358375831000, "i" : 5702 }, "optimeDate" : ISODate("2013-01-16T22:37:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 864, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 864, "optime" : { "t" : 1358375830000, "i" : 2946 }, "optimeDate" : ISODate("2013-01-16T22:37:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 874, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 862, "optime" : { "t" : 1358375827000, "i" : 4296 }, "optimeDate" : ISODate("2013-01-16T22:37:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 861, "optime" : { "t" : 1358375830000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-16T22:37:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 861, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 873, "optime" : { "t" : 1358375831000, "i" : 5702 }, "optimeDate" : ISODate("2013-01-16T22:37:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:37:12.109 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:37:13.356 [conn54] end connection 127.0.0.2:44591 (3 connections now open) m31000| Wed Jan 16 17:37:13.399 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:13.856 [initandlisten] connection accepted from 127.0.0.2:46611 #56 (4 connections now open) m31000| Wed Jan 16 17:37:15.399 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 879, "optime" : { "t" : 1358375835000, "i" : 7584 }, "optimeDate" : ISODate("2013-01-16T22:37:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 868, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 868, "optime" : { "t" : 1358375831000, "i" : 5702 }, "optimeDate" : ISODate("2013-01-16T22:37:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:36:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 878, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 866, "optime" : { "t" : 1358375831000, "i" : 5702 }, "optimeDate" : ISODate("2013-01-16T22:37:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 865, "optime" : { "t" : 1358375831000, "i" : 5702 }, "optimeDate" : ISODate("2013-01-16T22:37:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 865, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 877, "optime" : { "t" : 1358375835000, "i" : 7377 }, "optimeDate" : ISODate("2013-01-16T22:37:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:37:17.110 [conn53] end connection 127.0.0.4:50938 (2 connections now open) m31000| Wed Jan 16 17:37:17.400 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:17.610 [initandlisten] connection accepted from 127.0.0.4:49668 #55 (3 connections now open) m31002| Wed Jan 16 17:37:17.610 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:37:18.009 [conn55] end connection 127.0.0.3:60111 (3 connections now open) m31002| Wed Jan 16 17:37:18.009 [initandlisten] connection accepted from 127.0.0.3:58109 #57 (4 connections now open) m31000| Wed Jan 16 17:37:19.400 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 883, "optime" : { "t" : 1358375839000, "i" : 8583 }, "optimeDate" : ISODate("2013-01-16T22:37:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 872, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 872, "optime" : { "t" : 1358375838000, "i" : 41 }, "optimeDate" : ISODate("2013-01-16T22:37:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 882, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 870, "optime" : { "t" : 1358375835000, "i" : 7584 }, "optimeDate" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 869, "optime" : { "t" : 1358375835000, "i" : 7584 }, "optimeDate" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 869, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 881, "optime" : { "t" : 1358375839000, "i" : 8583 }, "optimeDate" : ISODate("2013-01-16T22:37:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:21.400 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:22.611 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:23.205 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:400147 200ms m31000| Wed Jan 16 17:37:23.400 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 888, "optime" : { "t" : 1358375844000, "i" : 563 }, "optimeDate" : ISODate("2013-01-16T22:37:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 877, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 877, "optime" : { "t" : 1358375843000, "i" : 8169 }, "optimeDate" : ISODate("2013-01-16T22:37:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 887, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 875, "optime" : { "t" : 1358375844000, "i" : 87 }, "optimeDate" : ISODate("2013-01-16T22:37:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 874, "optime" : { "t" : 1358375839000, "i" : 8583 }, "optimeDate" : ISODate("2013-01-16T22:37:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 874, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 886, "optime" : { "t" : 1358375844000, "i" : 534 }, "optimeDate" : ISODate("2013-01-16T22:37:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:25.401 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:37:27.401 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:27.611 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 892, "optime" : { "t" : 1358375848000, "i" : 1170 }, "optimeDate" : ISODate("2013-01-16T22:37:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 881, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 881, "optime" : { "t" : 1358375844000, "i" : 563 }, "optimeDate" : ISODate("2013-01-16T22:37:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 891, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 879, "optime" : { "t" : 1358375848000, "i" : 17 }, "optimeDate" : ISODate("2013-01-16T22:37:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 878, "optime" : { "t" : 1358375844000, "i" : 563 }, "optimeDate" : ISODate("2013-01-16T22:37:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 878, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 890, "optime" : { "t" : 1358375848000, "i" : 1170 }, "optimeDate" : ISODate("2013-01-16T22:37:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:37:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:37:29.401 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:31.402 [conn54] end connection 127.0.0.2:43980 (2 connections now open) m31001| Wed Jan 16 17:37:31.402 [initandlisten] connection accepted from 127.0.0.2:55311 #56 (3 connections now open) m31000| Wed Jan 16 17:37:31.402 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 896, "optime" : { "t" : 1358375852000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:37:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 885, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 885, "optime" : { "t" : 1358375851000, "i" : 1845 }, "optimeDate" : ISODate("2013-01-16T22:37:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 895, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 883, "optime" : { "t" : 1358375851000, "i" : 7072 }, "optimeDate" : ISODate("2013-01-16T22:37:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 882, "optime" : { "t" : 1358375848000, "i" : 1170 }, "optimeDate" : ISODate("2013-01-16T22:37:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 882, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 894, "optime" : { "t" : 1358375852000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:37:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:37:32.612 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:33.402 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:37:34.645 [conn40] end connection 127.0.0.4:33869 (4 connections now open) m31000| Wed Jan 16 17:37:34.645 [initandlisten] connection accepted from 127.0.0.4:42010 #41 (5 connections now open) m31000| Wed Jan 16 17:37:35.403 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 900, "optime" : { "t" : 1358375856000, "i" : 4213 }, "optimeDate" : ISODate("2013-01-16T22:37:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 889, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 889, "optime" : { "t" : 1358375856000, "i" : 2552 }, "optimeDate" : ISODate("2013-01-16T22:37:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 899, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 887, "optime" : { "t" : 1358375855000, "i" : 5786 }, "optimeDate" : ISODate("2013-01-16T22:37:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 886, "optime" : { "t" : 1358375852000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:37:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 886, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 898, "optime" : { "t" : 1358375856000, "i" : 3767 }, "optimeDate" : ISODate("2013-01-16T22:37:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:37.403 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:37.613 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:37:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:37:39.403 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 904, "optime" : { "t" : 1358375860000, "i" : 5637 }, "optimeDate" : ISODate("2013-01-16T22:37:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 893, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 893, "optime" : { "t" : 1358375856000, "i" : 4213 }, "optimeDate" : ISODate("2013-01-16T22:37:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 903, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 891, "optime" : { "t" : 1358375859000, "i" : 4362 }, "optimeDate" : ISODate("2013-01-16T22:37:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 890, "optime" : { "t" : 1358375856000, "i" : 4213 }, "optimeDate" : ISODate("2013-01-16T22:37:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 890, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 902, "optime" : { "t" : 1358375860000, "i" : 5591 }, "optimeDate" : ISODate("2013-01-16T22:37:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:41.404 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:42.613 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:43.404 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 908, "optime" : { "t" : 1358375864000, "i" : 6251 }, "optimeDate" : ISODate("2013-01-16T22:37:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 897, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 897, "optime" : { "t" : 1358375863000, "i" : 2094 }, "optimeDate" : ISODate("2013-01-16T22:37:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 907, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 895, "optime" : { "t" : 1358375864000, "i" : 80 }, "optimeDate" : ISODate("2013-01-16T22:37:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 894, "optime" : { "t" : 1358375864000, "i" : 5321 }, "optimeDate" : ISODate("2013-01-16T22:37:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 894, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 906, "optime" : { "t" : 1358375864000, "i" : 6251 }, "optimeDate" : ISODate("2013-01-16T22:37:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:45.405 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:37:47.405 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:47.614 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:47.907 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:245148 122ms m31002| Wed Jan 16 17:37:48.014 [conn57] end connection 127.0.0.3:58109 (3 connections now open) m31002| Wed Jan 16 17:37:48.014 [initandlisten] connection accepted from 127.0.0.3:34567 #58 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 912, "optime" : { "t" : 1358375868000, "i" : 7912 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 901, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 901, "optime" : { "t" : 1358375868000, "i" : 7664 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 911, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 899, "optime" : { "t" : 1358375868000, "i" : 16 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 898, "optime" : { "t" : 1358375868000, "i" : 6232 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 898, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 910, "optime" : { "t" : 1358375868000, "i" : 7664 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:49.405 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:51.361 [conn56] end connection 127.0.0.2:46611 (3 connections now open) m31000| Wed Jan 16 17:37:51.406 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:51.861 [initandlisten] connection accepted from 127.0.0.2:33405 #59 (4 connections now open) m31002| Wed Jan 16 17:37:52.614 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 917, "optime" : { "t" : 1358375873000, "i" : 333 }, "optimeDate" : ISODate("2013-01-16T22:37:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 906, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 906, "optime" : { "t" : 1358375868000, "i" : 7912 }, "optimeDate" : ISODate("2013-01-16T22:37:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 916, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 904, "optime" : { "t" : 1358375872000, "i" : 313 }, "optimeDate" : ISODate("2013-01-16T22:37:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 903, "optime" : { "t" : 1358375872000, "i" : 5808 }, "optimeDate" : ISODate("2013-01-16T22:37:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 903, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 915, "optime" : { "t" : 1358375873000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:37:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:53.406 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:55.115 [conn55] end connection 127.0.0.4:49668 (2 connections now open) m31000| Wed Jan 16 17:37:55.407 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:37:55.615 [initandlisten] connection accepted from 127.0.0.4:45015 #57 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 921, "optime" : { "t" : 1358375877000, "i" : 1987 }, "optimeDate" : ISODate("2013-01-16T22:37:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 910, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 910, "optime" : { "t" : 1358375876000, "i" : 6808 }, "optimeDate" : ISODate("2013-01-16T22:37:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 920, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 908, "optime" : { "t" : 1358375873000, "i" : 333 }, "optimeDate" : ISODate("2013-01-16T22:37:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:37:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 907, "optime" : { "t" : 1358375876000, "i" : 4970 }, "optimeDate" : ISODate("2013-01-16T22:37:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:56Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 907, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:37:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 919, "optime" : { "t" : 1358375877000, "i" : 1987 }, "optimeDate" : ISODate("2013-01-16T22:37:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:37:57.407 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:37:58.116 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:37:59.407 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:38:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:38:01.408 [conn56] end connection 127.0.0.2:55311 (2 connections now open) m31001| Wed Jan 16 17:38:01.408 [initandlisten] connection accepted from 127.0.0.2:53125 #58 (3 connections now open) m31000| Wed Jan 16 17:38:01.408 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 925, "optime" : { "t" : 1358375881000, "i" : 3568 }, "optimeDate" : ISODate("2013-01-16T22:38:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 914, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 914, "optime" : { "t" : 1358375877000, "i" : 1987 }, "optimeDate" : ISODate("2013-01-16T22:37:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 924, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 912, "optime" : { "t" : 1358375877000, "i" : 1987 }, "optimeDate" : ISODate("2013-01-16T22:37:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 911, "optime" : { "t" : 1358375880000, "i" : 3459 }, "optimeDate" : ISODate("2013-01-16T22:38:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 911, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 923, "optime" : { "t" : 1358375881000, "i" : 3568 }, "optimeDate" : ISODate("2013-01-16T22:38:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:38:03.116 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:03.408 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:04.651 [conn41] end connection 127.0.0.4:42010 (4 connections now open) m31000| Wed Jan 16 17:38:04.651 [initandlisten] connection accepted from 127.0.0.4:54084 #42 (5 connections now open) m31000| Wed Jan 16 17:38:05.409 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 929, "optime" : { "t" : 1358375885000, "i" : 5196 }, "optimeDate" : ISODate("2013-01-16T22:38:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 918, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 918, "optime" : { "t" : 1358375881000, "i" : 3568 }, "optimeDate" : ISODate("2013-01-16T22:38:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 928, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 916, "optime" : { "t" : 1358375881000, "i" : 3568 }, "optimeDate" : ISODate("2013-01-16T22:38:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:05Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 915, "optime" : { "t" : 1358375884000, "i" : 1597 }, "optimeDate" : ISODate("2013-01-16T22:38:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 915, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 927, "optime" : { "t" : 1358375885000, "i" : 5196 }, "optimeDate" : ISODate("2013-01-16T22:38:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:07.409 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:08.117 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:09.409 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 933, "optime" : { "t" : 1358375889000, "i" : 6923 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 922, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 922, "optime" : { "t" : 1358375889000, "i" : 2747 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 932, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 920, "optime" : { "t" : 1358375885000, "i" : 5196 }, "optimeDate" : ISODate("2013-01-16T22:38:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 919, "optime" : { "t" : 1358375885000, "i" : 5196 }, "optimeDate" : ISODate("2013-01-16T22:38:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 919, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 931, "optime" : { "t" : 1358375889000, "i" : 6923 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:38:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:38:11.410 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:13.118 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:13.410 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 937, "optime" : { "t" : 1358375893000, "i" : 8810 }, "optimeDate" : ISODate("2013-01-16T22:38:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 926, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 926, "optime" : { "t" : 1358375889000, "i" : 6923 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 936, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 924, "optime" : { "t" : 1358375889000, "i" : 6923 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 923, "optime" : { "t" : 1358375889000, "i" : 6923 }, "optimeDate" : ISODate("2013-01-16T22:38:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 923, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 935, "optime" : { "t" : 1358375893000, "i" : 8109 }, "optimeDate" : ISODate("2013-01-16T22:38:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:15.410 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:17.411 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:18.019 [conn58] end connection 127.0.0.3:34567 (3 connections now open) m31002| Wed Jan 16 17:38:18.019 [initandlisten] connection accepted from 127.0.0.3:53974 #60 (4 connections now open) m31002| Wed Jan 16 17:38:18.118 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 942, "optime" : { "t" : 1358375898000, "i" : 1357 }, "optimeDate" : ISODate("2013-01-16T22:38:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 931, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 931, "optime" : { "t" : 1358375893000, "i" : 8810 }, "optimeDate" : ISODate("2013-01-16T22:38:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:37:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 941, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 929, "optime" : { "t" : 1358375897000, "i" : 8445 }, "optimeDate" : ISODate("2013-01-16T22:38:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 928, "optime" : { "t" : 1358375893000, "i" : 8810 }, "optimeDate" : ISODate("2013-01-16T22:38:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 928, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 940, "optime" : { "t" : 1358375898000, "i" : 982 }, "optimeDate" : ISODate("2013-01-16T22:38:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:19.411 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:21.411 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 946, "optime" : { "t" : 1358375902000, "i" : 3044 }, "optimeDate" : ISODate("2013-01-16T22:38:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 935, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 935, "optime" : { "t" : 1358375901000, "i" : 5504 }, "optimeDate" : ISODate("2013-01-16T22:38:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 945, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 933, "optime" : { "t" : 1358375901000, "i" : 6955 }, "optimeDate" : ISODate("2013-01-16T22:38:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 932, "optime" : { "t" : 1358375898000, "i" : 1357 }, "optimeDate" : ISODate("2013-01-16T22:38:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 932, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 944, "optime" : { "t" : 1358375902000, "i" : 2936 }, "optimeDate" : ISODate("2013-01-16T22:38:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:38:23.119 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:23.412 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:25.412 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 950, "optime" : { "t" : 1358375906000, "i" : 4925 }, "optimeDate" : ISODate("2013-01-16T22:38:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 939, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 939, "optime" : { "t" : 1358375902000, "i" : 3044 }, "optimeDate" : ISODate("2013-01-16T22:38:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 949, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 937, "optime" : { "t" : 1358375905000, "i" : 5075 }, "optimeDate" : ISODate("2013-01-16T22:38:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 936, "optime" : { "t" : 1358375902000, "i" : 3044 }, "optimeDate" : ISODate("2013-01-16T22:38:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 936, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 948, "optime" : { "t" : 1358375906000, "i" : 4643 }, "optimeDate" : ISODate("2013-01-16T22:38:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:27.412 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:28.119 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:38:29.367 [conn59] end connection 127.0.0.2:33405 (3 connections now open) m31000| Wed Jan 16 17:38:29.413 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:29.867 [initandlisten] connection accepted from 127.0.0.2:60567 #61 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 954, "optime" : { "t" : 1358375910000, "i" : 5783 }, "optimeDate" : ISODate("2013-01-16T22:38:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 943, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 943, "optime" : { "t" : 1358375909000, "i" : 2975 }, "optimeDate" : ISODate("2013-01-16T22:38:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 953, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 941, "optime" : { "t" : 1358375910000, "i" : 132 }, "optimeDate" : ISODate("2013-01-16T22:38:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 940, "optime" : { "t" : 1358375910000, "i" : 5445 }, "optimeDate" : ISODate("2013-01-16T22:38:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 940, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 952, "optime" : { "t" : 1358375910000, "i" : 5445 }, "optimeDate" : ISODate("2013-01-16T22:38:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:38:31.413 [conn58] end connection 127.0.0.2:53125 (2 connections now open) m31001| Wed Jan 16 17:38:31.413 [initandlisten] connection accepted from 127.0.0.2:52489 #59 (3 connections now open) m31000| Wed Jan 16 17:38:31.413 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:38:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:38:33.120 [conn57] end connection 127.0.0.4:45015 (2 connections now open) m31000| Wed Jan 16 17:38:33.414 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:38:33.620 [initandlisten] connection accepted from 127.0.0.4:49184 #60 (3 connections now open) m31002| Wed Jan 16 17:38:33.620 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:34.657 [conn42] end connection 127.0.0.4:54084 (4 connections now open) m31000| Wed Jan 16 17:38:34.657 [initandlisten] connection accepted from 127.0.0.4:51278 #43 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 958, "optime" : { "t" : 1358375914000, "i" : 7263 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 947, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 947, "optime" : { "t" : 1358375914000, "i" : 6910 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 957, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 945, "optime" : { "t" : 1358375914000, "i" : 110 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 944, "optime" : { "t" : 1358375914000, "i" : 5584 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 944, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 956, "optime" : { "t" : 1358375914000, "i" : 6910 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:35.414 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:37.415 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:38.621 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 963, "optime" : { "t" : 1358375919000, "i" : 539 }, "optimeDate" : ISODate("2013-01-16T22:38:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 952, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 952, "optime" : { "t" : 1358375914000, "i" : 7263 }, "optimeDate" : ISODate("2013-01-16T22:38:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 962, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 950, "optime" : { "t" : 1358375918000, "i" : 182 }, "optimeDate" : ISODate("2013-01-16T22:38:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 949, "optime" : { "t" : 1358375918000, "i" : 5662 }, "optimeDate" : ISODate("2013-01-16T22:38:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 949, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 961, "optime" : { "t" : 1358375919000, "i" : 539 }, "optimeDate" : ISODate("2013-01-16T22:38:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:39.415 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:41.415 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:38:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 967, "optime" : { "t" : 1358375923000, "i" : 1972 }, "optimeDate" : ISODate("2013-01-16T22:38:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 956, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 956, "optime" : { "t" : 1358375922000, "i" : 2497 }, "optimeDate" : ISODate("2013-01-16T22:38:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 966, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 954, "optime" : { "t" : 1358375919000, "i" : 539 }, "optimeDate" : ISODate("2013-01-16T22:38:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 953, "optime" : { "t" : 1358375922000, "i" : 5132 }, "optimeDate" : ISODate("2013-01-16T22:38:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 953, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 965, "optime" : { "t" : 1358375923000, "i" : 1972 }, "optimeDate" : ISODate("2013-01-16T22:38:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:43.416 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:43.622 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:45.416 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 971, "optime" : { "t" : 1358375927000, "i" : 2950 }, "optimeDate" : ISODate("2013-01-16T22:38:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 960, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 960, "optime" : { "t" : 1358375923000, "i" : 1972 }, "optimeDate" : ISODate("2013-01-16T22:38:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 970, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 958, "optime" : { "t" : 1358375923000, "i" : 1972 }, "optimeDate" : ISODate("2013-01-16T22:38:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 957, "optime" : { "t" : 1358375926000, "i" : 3925 }, "optimeDate" : ISODate("2013-01-16T22:38:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 957, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 969, "optime" : { "t" : 1358375927000, "i" : 2829 }, "optimeDate" : ISODate("2013-01-16T22:38:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:47.417 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:48.025 [conn60] end connection 127.0.0.3:53974 (3 connections now open) m31002| Wed Jan 16 17:38:48.025 [initandlisten] connection accepted from 127.0.0.3:37113 #62 (4 connections now open) m31002| Wed Jan 16 17:38:48.622 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:49.417 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:38:51.417 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 975, "optime" : { "t" : 1358375931000, "i" : 4262 }, "optimeDate" : ISODate("2013-01-16T22:38:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 964, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 964, "optime" : { "t" : 1358375927000, "i" : 2950 }, "optimeDate" : ISODate("2013-01-16T22:38:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 974, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 962, "optime" : { "t" : 1358375927000, "i" : 2950 }, "optimeDate" : ISODate("2013-01-16T22:38:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:51Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 961, "optime" : { "t" : 1358375930000, "i" : 2843 }, "optimeDate" : ISODate("2013-01-16T22:38:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 961, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 973, "optime" : { "t" : 1358375931000, "i" : 4262 }, "optimeDate" : ISODate("2013-01-16T22:38:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:53.418 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:53.623 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:55.418 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 979, "optime" : { "t" : 1358375935000, "i" : 5214 }, "optimeDate" : ISODate("2013-01-16T22:38:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 968, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 968, "optime" : { "t" : 1358375934000, "i" : 3566 }, "optimeDate" : ISODate("2013-01-16T22:38:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 978, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 966, "optime" : { "t" : 1358375931000, "i" : 4262 }, "optimeDate" : ISODate("2013-01-16T22:38:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 965, "optime" : { "t" : 1358375934000, "i" : 1245 }, "optimeDate" : ISODate("2013-01-16T22:38:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 965, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 977, "optime" : { "t" : 1358375935000, "i" : 5214 }, "optimeDate" : ISODate("2013-01-16T22:38:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:38:57.418 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:38:58.624 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:38:59.419 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 983, "optime" : { "t" : 1358375939000, "i" : 6675 }, "optimeDate" : ISODate("2013-01-16T22:38:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 972, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 972, "optime" : { "t" : 1358375935000, "i" : 5214 }, "optimeDate" : ISODate("2013-01-16T22:38:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 982, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 970, "optime" : { "t" : 1358375935000, "i" : 5214 }, "optimeDate" : ISODate("2013-01-16T22:38:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:38:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:38:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 969, "optime" : { "t" : 1358375938000, "i" : 404 }, "optimeDate" : ISODate("2013-01-16T22:38:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 969, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 981, "optime" : { "t" : 1358375939000, "i" : 6675 }, "optimeDate" : ISODate("2013-01-16T22:38:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:39:01.419 [conn59] end connection 127.0.0.2:52489 (2 connections now open) m31001| Wed Jan 16 17:39:01.419 [initandlisten] connection accepted from 127.0.0.2:46157 #61 (3 connections now open) m31000| Wed Jan 16 17:39:01.419 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:03.420 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:03.624 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 987, "optime" : { "t" : 1358375943000, "i" : 8354 }, "optimeDate" : ISODate("2013-01-16T22:39:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 976, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 976, "optime" : { "t" : 1358375939000, "i" : 6675 }, "optimeDate" : ISODate("2013-01-16T22:38:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 986, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 974, "optime" : { "t" : 1358375939000, "i" : 6675 }, "optimeDate" : ISODate("2013-01-16T22:38:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:03Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 973, "optime" : { "t" : 1358375939000, "i" : 6675 }, "optimeDate" : ISODate("2013-01-16T22:38:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 973, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 985, "optime" : { "t" : 1358375943000, "i" : 8354 }, "optimeDate" : ISODate("2013-01-16T22:39:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:04.661 [conn43] end connection 127.0.0.4:51278 (4 connections now open) m31000| Wed Jan 16 17:39:04.661 [initandlisten] connection accepted from 127.0.0.4:58237 #44 (5 connections now open) m31001| Wed Jan 16 17:39:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:39:05.420 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:07.372 [conn61] end connection 127.0.0.2:60567 (3 connections now open) m31000| Wed Jan 16 17:39:07.420 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:07.872 [initandlisten] connection accepted from 127.0.0.2:39535 #63 (4 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:39:08.215 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:781223 390ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 992, "optime" : { "t" : 1358375948000, "i" : 2139 }, "optimeDate" : ISODate("2013-01-16T22:39:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 981, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 981, "optime" : { "t" : 1358375947000, "i" : 6881 }, "optimeDate" : ISODate("2013-01-16T22:39:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 991, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 979, "optime" : { "t" : 1358375947000, "i" : 6894 }, "optimeDate" : ISODate("2013-01-16T22:39:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 978, "optime" : { "t" : 1358375943000, "i" : 8354 }, "optimeDate" : ISODate("2013-01-16T22:39:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 978, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:08Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 990, "optime" : { "t" : 1358375948000, "i" : 2139 }, "optimeDate" : ISODate("2013-01-16T22:39:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:39:08.625 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:39:09.421 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:39:11.125 [conn60] end connection 127.0.0.4:49184 (2 connections now open) m31000| Wed Jan 16 17:39:11.421 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:39:11.625 [initandlisten] connection accepted from 127.0.0.4:50388 #62 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 996, "optime" : { "t" : 1358375952000, "i" : 3902 }, "optimeDate" : ISODate("2013-01-16T22:39:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 985, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 985, "optime" : { "t" : 1358375948000, "i" : 2139 }, "optimeDate" : ISODate("2013-01-16T22:39:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 995, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 983, "optime" : { "t" : 1358375951000, "i" : 6089 }, "optimeDate" : ISODate("2013-01-16T22:39:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 982, "optime" : { "t" : 1358375948000, "i" : 2139 }, "optimeDate" : ISODate("2013-01-16T22:39:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 982, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 994, "optime" : { "t" : 1358375952000, "i" : 3522 }, "optimeDate" : ISODate("2013-01-16T22:39:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:13.421 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:14.126 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:39:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:39:15.422 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1000, "optime" : { "t" : 1358375956000, "i" : 4434 }, "optimeDate" : ISODate("2013-01-16T22:39:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 989, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 989, "optime" : { "t" : 1358375952000, "i" : 3902 }, "optimeDate" : ISODate("2013-01-16T22:39:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 999, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 987, "optime" : { "t" : 1358375955000, "i" : 5473 }, "optimeDate" : ISODate("2013-01-16T22:39:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 986, "optime" : { "t" : 1358375952000, "i" : 3902 }, "optimeDate" : ISODate("2013-01-16T22:39:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 986, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 998, "optime" : { "t" : 1358375956000, "i" : 4434 }, "optimeDate" : ISODate("2013-01-16T22:39:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:17.422 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:18.031 [conn62] end connection 127.0.0.3:37113 (3 connections now open) m31002| Wed Jan 16 17:39:18.031 [initandlisten] connection accepted from 127.0.0.3:54823 #64 (4 connections now open) m31002| Wed Jan 16 17:39:19.126 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:39:19.423 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1004, "optime" : { "t" : 1358375960000, "i" : 6157 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 993, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 993, "optime" : { "t" : 1358375960000, "i" : 3276 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:38:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1003, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 991, "optime" : { "t" : 1358375960000, "i" : 199 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 990, "optime" : { "t" : 1358375960000, "i" : 5928 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 990, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1002, "optime" : { "t" : 1358375960000, "i" : 6151 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:21.423 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:23.423 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:24.127 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1009, "optime" : { "t" : 1358375965000, "i" : 60 }, "optimeDate" : ISODate("2013-01-16T22:39:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 998, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 998, "optime" : { "t" : 1358375960000, "i" : 6157 }, "optimeDate" : ISODate("2013-01-16T22:39:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1008, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 996, "optime" : { "t" : 1358375964000, "i" : 491 }, "optimeDate" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:24Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 995, "optime" : { "t" : 1358375964000, "i" : 5829 }, "optimeDate" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 995, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1007, "optime" : { "t" : 1358375965000, "i" : 60 }, "optimeDate" : ISODate("2013-01-16T22:39:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:25.424 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:27.424 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:29.128 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1013, "optime" : { "t" : 1358375969000, "i" : 1739 }, "optimeDate" : ISODate("2013-01-16T22:39:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1002, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1002, "optime" : { "t" : 1358375965000, "i" : 60 }, "optimeDate" : ISODate("2013-01-16T22:39:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1012, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1000, "optime" : { "t" : 1358375968000, "i" : 27 }, "optimeDate" : ISODate("2013-01-16T22:39:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:29Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 999, "optime" : { "t" : 1358375968000, "i" : 5492 }, "optimeDate" : ISODate("2013-01-16T22:39:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 999, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1011, "optime" : { "t" : 1358375969000, "i" : 1739 }, "optimeDate" : ISODate("2013-01-16T22:39:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:29.425 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:39:31.425 [conn61] end connection 127.0.0.2:46157 (2 connections now open) m31001| Wed Jan 16 17:39:31.425 [initandlisten] connection accepted from 127.0.0.2:41071 #63 (3 connections now open) m31000| Wed Jan 16 17:39:31.425 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:32.335 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.4, filling with zeroes... m31000| Wed Jan 16 17:39:33.426 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:34.128 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:39:34.357 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.4, size: 256MB, took 2.021 secs m31000| Wed Jan 16 17:39:34.357 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:2022171 2022ms m31002| Wed Jan 16 17:39:34.358 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.4, filling with zeroes... STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:39:34.667 [conn44] end connection 127.0.0.4:58237 (4 connections now open) m31000| Wed Jan 16 17:39:34.667 [initandlisten] connection accepted from 127.0.0.4:49205 #45 (5 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1018, "optime" : { "t" : 1358375974000, "i" : 9187 }, "optimeDate" : ISODate("2013-01-16T22:39:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1007, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1007, "optime" : { "t" : 1358375972000, "i" : 813 }, "optimeDate" : ISODate("2013-01-16T22:39:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1017, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1005, "optime" : { "t" : 1358375972000, "i" : 813 }, "optimeDate" : ISODate("2013-01-16T22:39:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1004, "optime" : { "t" : 1358375974000, "i" : 6582 }, "optimeDate" : ISODate("2013-01-16T22:39:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1004, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1016, "optime" : { "t" : 1358375972000, "i" : 813 }, "optimeDate" : ISODate("2013-01-16T22:39:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:35.426 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:36.381 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.4, size: 256MB, took 2.023 secs m31001| Wed Jan 16 17:39:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:39:37.427 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:39.129 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:39:39.427 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:40.316 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:5049079 2524ms m31000| Wed Jan 16 17:39:40.317 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:125383126206669 ntoreturn:0 keyUpdates:0 locks(micros) r:72 nreturned:1 reslen:117 5529ms m31000| Wed Jan 16 17:39:40.317 [conn11] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:129158504470451 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:117 nreturned:1 reslen:37 3556ms STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:39:41.427 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:41.980 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:3186939 1593ms m31000| Wed Jan 16 17:39:41.981 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:125383126206669 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:67 nreturned:2 reslen:214 1596ms m31000| Wed Jan 16 17:39:42.231 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:361299 180ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1026, "optime" : { "t" : 1358375982000, "i" : 7110 }, "optimeDate" : ISODate("2013-01-16T22:39:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1015, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1015, "optime" : { "t" : 1358375980000, "i" : 685 }, "optimeDate" : ISODate("2013-01-16T22:39:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1025, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1013, "optime" : { "t" : 1358375981000, "i" : 12 }, "optimeDate" : ISODate("2013-01-16T22:39:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1012, "optime" : { "t" : 1358375980000, "i" : 2337 }, "optimeDate" : ISODate("2013-01-16T22:39:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1012, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1024, "optime" : { "t" : 1358375982000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:39:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:43.428 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:44.130 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:39:45.377 [conn63] end connection 127.0.0.2:39535 (3 connections now open) m31000| Wed Jan 16 17:39:45.428 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:45.877 [initandlisten] connection accepted from 127.0.0.2:45395 #65 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1030, "optime" : { "t" : 1358375986000, "i" : 6777 }, "optimeDate" : ISODate("2013-01-16T22:39:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1019, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1019, "optime" : { "t" : 1358375985000, "i" : 1853 }, "optimeDate" : ISODate("2013-01-16T22:39:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1029, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1017, "optime" : { "t" : 1358375986000, "i" : 180 }, "optimeDate" : ISODate("2013-01-16T22:39:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1016, "optime" : { "t" : 1358375986000, "i" : 5495 }, "optimeDate" : ISODate("2013-01-16T22:39:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1016, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1028, "optime" : { "t" : 1358375986000, "i" : 6777 }, "optimeDate" : ISODate("2013-01-16T22:39:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:39:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:39:47.429 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:48.035 [conn64] end connection 127.0.0.3:54823 (3 connections now open) m31002| Wed Jan 16 17:39:48.036 [initandlisten] connection accepted from 127.0.0.3:39692 #66 (4 connections now open) m31001| Wed Jan 16 17:39:49.130 [conn62] end connection 127.0.0.4:50388 (2 connections now open) m31000| Wed Jan 16 17:39:49.429 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:39:49.630 [initandlisten] connection accepted from 127.0.0.4:45448 #64 (3 connections now open) m31002| Wed Jan 16 17:39:49.631 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1034, "optime" : { "t" : 1358375990000, "i" : 8318 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1023, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1023, "optime" : { "t" : 1358375990000, "i" : 7260 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1033, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1021, "optime" : { "t" : 1358375990000, "i" : 85 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1020, "optime" : { "t" : 1358375990000, "i" : 5924 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1020, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1032, "optime" : { "t" : 1358375990000, "i" : 8318 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:51.429 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:53.430 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:54.631 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1039, "optime" : { "t" : 1358375995000, "i" : 922 }, "optimeDate" : ISODate("2013-01-16T22:39:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1028, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1028, "optime" : { "t" : 1358375990000, "i" : 8318 }, "optimeDate" : ISODate("2013-01-16T22:39:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1038, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1026, "optime" : { "t" : 1358375994000, "i" : 59 }, "optimeDate" : ISODate("2013-01-16T22:39:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1025, "optime" : { "t" : 1358375994000, "i" : 5447 }, "optimeDate" : ISODate("2013-01-16T22:39:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1025, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1037, "optime" : { "t" : 1358375995000, "i" : 922 }, "optimeDate" : ISODate("2013-01-16T22:39:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:55.430 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:39:57.431 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1043, "optime" : { "t" : 1358375999000, "i" : 2171 }, "optimeDate" : ISODate("2013-01-16T22:39:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1032, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1032, "optime" : { "t" : 1358375998000, "i" : 2421 }, "optimeDate" : ISODate("2013-01-16T22:39:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1042, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1030, "optime" : { "t" : 1358375995000, "i" : 922 }, "optimeDate" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:39:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1029, "optime" : { "t" : 1358375998000, "i" : 4676 }, "optimeDate" : ISODate("2013-01-16T22:39:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1029, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:39:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1041, "optime" : { "t" : 1358375999000, "i" : 2082 }, "optimeDate" : ISODate("2013-01-16T22:39:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:39:59.431 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:39:59.632 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:40:01.431 [conn63] end connection 127.0.0.2:41071 (2 connections now open) m31001| Wed Jan 16 17:40:01.432 [initandlisten] connection accepted from 127.0.0.2:47668 #65 (3 connections now open) m31000| Wed Jan 16 17:40:01.432 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1047, "optime" : { "t" : 1358376003000, "i" : 3603 }, "optimeDate" : ISODate("2013-01-16T22:40:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1036, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1036, "optime" : { "t" : 1358375999000, "i" : 2171 }, "optimeDate" : ISODate("2013-01-16T22:39:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1046, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1034, "optime" : { "t" : 1358375999000, "i" : 2171 }, "optimeDate" : ISODate("2013-01-16T22:39:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1033, "optime" : { "t" : 1358376002000, "i" : 3104 }, "optimeDate" : ISODate("2013-01-16T22:40:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1033, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1045, "optime" : { "t" : 1358376003000, "i" : 3185 }, "optimeDate" : ISODate("2013-01-16T22:40:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:03.432 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:04.633 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:04.672 [conn45] end connection 127.0.0.4:49205 (4 connections now open) m31000| Wed Jan 16 17:40:04.672 [initandlisten] connection accepted from 127.0.0.4:59733 #46 (5 connections now open) m31000| Wed Jan 16 17:40:05.432 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1051, "optime" : { "t" : 1358376007000, "i" : 3827 }, "optimeDate" : ISODate("2013-01-16T22:40:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1040, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1040, "optime" : { "t" : 1358376003000, "i" : 3603 }, "optimeDate" : ISODate("2013-01-16T22:40:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1050, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1038, "optime" : { "t" : 1358376003000, "i" : 3603 }, "optimeDate" : ISODate("2013-01-16T22:40:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:07Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1037, "optime" : { "t" : 1358376006000, "i" : 3157 }, "optimeDate" : ISODate("2013-01-16T22:40:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1037, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1049, "optime" : { "t" : 1358376007000, "i" : 3060 }, "optimeDate" : ISODate("2013-01-16T22:40:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:07.433 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:40:08.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:40:09.433 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:09.633 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:11.434 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1055, "optime" : { "t" : 1358376011000, "i" : 4865 }, "optimeDate" : ISODate("2013-01-16T22:40:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1044, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1044, "optime" : { "t" : 1358376010000, "i" : 3991 }, "optimeDate" : ISODate("2013-01-16T22:40:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1054, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1042, "optime" : { "t" : 1358376007000, "i" : 3827 }, "optimeDate" : ISODate("2013-01-16T22:40:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1041, "optime" : { "t" : 1358376010000, "i" : 2312 }, "optimeDate" : ISODate("2013-01-16T22:40:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1041, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1053, "optime" : { "t" : 1358376011000, "i" : 4821 }, "optimeDate" : ISODate("2013-01-16T22:40:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:13.434 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:14.634 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:15.434 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1059, "optime" : { "t" : 1358376015000, "i" : 6435 }, "optimeDate" : ISODate("2013-01-16T22:40:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1048, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1048, "optime" : { "t" : 1358376011000, "i" : 4865 }, "optimeDate" : ISODate("2013-01-16T22:40:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1058, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1046, "optime" : { "t" : 1358376011000, "i" : 4865 }, "optimeDate" : ISODate("2013-01-16T22:40:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1045, "optime" : { "t" : 1358376014000, "i" : 257 }, "optimeDate" : ISODate("2013-01-16T22:40:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1045, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1057, "optime" : { "t" : 1358376015000, "i" : 5813 }, "optimeDate" : ISODate("2013-01-16T22:40:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:17.435 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:18.041 [conn66] end connection 127.0.0.3:39692 (3 connections now open) m31002| Wed Jan 16 17:40:18.041 [initandlisten] connection accepted from 127.0.0.3:44564 #67 (4 connections now open) m31001| Wed Jan 16 17:40:18.858 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:40:19.435 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:19.635 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1063, "optime" : { "t" : 1358376019000, "i" : 7743 }, "optimeDate" : ISODate("2013-01-16T22:40:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1052, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1052, "optime" : { "t" : 1358376015000, "i" : 6435 }, "optimeDate" : ISODate("2013-01-16T22:40:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1062, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1050, "optime" : { "t" : 1358376015000, "i" : 6435 }, "optimeDate" : ISODate("2013-01-16T22:40:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:19Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1049, "optime" : { "t" : 1358376015000, "i" : 6435 }, "optimeDate" : ISODate("2013-01-16T22:40:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1049, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1061, "optime" : { "t" : 1358376019000, "i" : 7743 }, "optimeDate" : ISODate("2013-01-16T22:40:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:21.435 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:23.383 [conn65] end connection 127.0.0.2:45395 (3 connections now open) m31000| Wed Jan 16 17:40:23.436 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:23.883 [initandlisten] connection accepted from 127.0.0.2:32864 #68 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1068, "optime" : { "t" : 1358376024000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:40:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1057, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1057, "optime" : { "t" : 1358376023000, "i" : 7821 }, "optimeDate" : ISODate("2013-01-16T22:40:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1067, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1055, "optime" : { "t" : 1358376024000, "i" : 298 }, "optimeDate" : ISODate("2013-01-16T22:40:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1054, "optime" : { "t" : 1358376019000, "i" : 7743 }, "optimeDate" : ISODate("2013-01-16T22:40:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1054, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1066, "optime" : { "t" : 1358376024000, "i" : 298 }, "optimeDate" : ISODate("2013-01-16T22:40:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:40:24.635 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:25.436 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:40:27.136 [conn64] end connection 127.0.0.4:45448 (2 connections now open) m31000| Wed Jan 16 17:40:27.436 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:40:27.636 [initandlisten] connection accepted from 127.0.0.4:39190 #66 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1072, "optime" : { "t" : 1358376028000, "i" : 2161 }, "optimeDate" : ISODate("2013-01-16T22:40:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1061, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1061, "optime" : { "t" : 1358376024000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:40:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:39:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1071, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1059, "optime" : { "t" : 1358376027000, "i" : 7838 }, "optimeDate" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1058, "optime" : { "t" : 1358376024000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:40:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1058, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1070, "optime" : { "t" : 1358376028000, "i" : 2161 }, "optimeDate" : ISODate("2013-01-16T22:40:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:29.437 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:30.136 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:40:31.437 [conn65] end connection 127.0.0.2:47668 (2 connections now open) m31001| Wed Jan 16 17:40:31.437 [initandlisten] connection accepted from 127.0.0.2:38518 #67 (3 connections now open) m31000| Wed Jan 16 17:40:31.437 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1076, "optime" : { "t" : 1358376032000, "i" : 3113 }, "optimeDate" : ISODate("2013-01-16T22:40:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1065, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1065, "optime" : { "t" : 1358376031000, "i" : 745 }, "optimeDate" : ISODate("2013-01-16T22:40:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1075, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1063, "optime" : { "t" : 1358376032000, "i" : 327 }, "optimeDate" : ISODate("2013-01-16T22:40:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1062, "optime" : { "t" : 1358376028000, "i" : 2161 }, "optimeDate" : ISODate("2013-01-16T22:40:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1062, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1074, "optime" : { "t" : 1358376032000, "i" : 3113 }, "optimeDate" : ISODate("2013-01-16T22:40:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:33.438 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:40:34.678 [conn46] end connection 127.0.0.4:59733 (4 connections now open) m31000| Wed Jan 16 17:40:34.678 [initandlisten] connection accepted from 127.0.0.4:47780 #47 (5 connections now open) m31002| Wed Jan 16 17:40:35.137 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:35.438 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1080, "optime" : { "t" : 1358376036000, "i" : 4354 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1069, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1069, "optime" : { "t" : 1358376036000, "i" : 2959 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1079, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1067, "optime" : { "t" : 1358376036000, "i" : 229 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1066, "optime" : { "t" : 1358376032000, "i" : 3113 }, "optimeDate" : ISODate("2013-01-16T22:40:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1066, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1078, "optime" : { "t" : 1358376036000, "i" : 4354 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:37.438 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:40:39.439 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:40.137 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1084, "optime" : { "t" : 1358376040000, "i" : 5919 }, "optimeDate" : ISODate("2013-01-16T22:40:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1073, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1073, "optime" : { "t" : 1358376036000, "i" : 4354 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1083, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1071, "optime" : { "t" : 1358376040000, "i" : 104 }, "optimeDate" : ISODate("2013-01-16T22:40:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1070, "optime" : { "t" : 1358376036000, "i" : 4354 }, "optimeDate" : ISODate("2013-01-16T22:40:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1070, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1082, "optime" : { "t" : 1358376040000, "i" : 5428 }, "optimeDate" : ISODate("2013-01-16T22:40:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:40:40.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:40:41.439 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:40:43.440 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1088, "optime" : { "t" : 1358376044000, "i" : 6706 }, "optimeDate" : ISODate("2013-01-16T22:40:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1077, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1077, "optime" : { "t" : 1358376043000, "i" : 1743 }, "optimeDate" : ISODate("2013-01-16T22:40:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1087, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1075, "optime" : { "t" : 1358376044000, "i" : 70 }, "optimeDate" : ISODate("2013-01-16T22:40:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1074, "optime" : { "t" : 1358376044000, "i" : 5872 }, "optimeDate" : ISODate("2013-01-16T22:40:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1074, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1086, "optime" : { "t" : 1358376044000, "i" : 6281 }, "optimeDate" : ISODate("2013-01-16T22:40:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:40:45.138 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:45.440 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:40:47.440 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:48.046 [conn67] end connection 127.0.0.3:44564 (3 connections now open) m31002| Wed Jan 16 17:40:48.046 [initandlisten] connection accepted from 127.0.0.3:49436 #69 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1092, "optime" : { "t" : 1358376048000, "i" : 8324 }, "optimeDate" : ISODate("2013-01-16T22:40:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1081, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1081, "optime" : { "t" : 1358376044000, "i" : 6706 }, "optimeDate" : ISODate("2013-01-16T22:40:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1091, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1079, "optime" : { "t" : 1358376048000, "i" : 815 }, "optimeDate" : ISODate("2013-01-16T22:40:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1078, "optime" : { "t" : 1358376048000, "i" : 6636 }, "optimeDate" : ISODate("2013-01-16T22:40:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1078, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1090, "optime" : { "t" : 1358376048000, "i" : 8206 }, "optimeDate" : ISODate("2013-01-16T22:40:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:49.441 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:50.139 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:40:50.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:40:51.441 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1097, "optime" : { "t" : 1358376053000, "i" : 894 }, "optimeDate" : ISODate("2013-01-16T22:40:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1086, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1086, "optime" : { "t" : 1358376048000, "i" : 8324 }, "optimeDate" : ISODate("2013-01-16T22:40:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1096, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1084, "optime" : { "t" : 1358376052000, "i" : 27 }, "optimeDate" : ISODate("2013-01-16T22:40:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1083, "optime" : { "t" : 1358376052000, "i" : 5800 }, "optimeDate" : ISODate("2013-01-16T22:40:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1083, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1095, "optime" : { "t" : 1358376053000, "i" : 894 }, "optimeDate" : ISODate("2013-01-16T22:40:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:53.441 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:40:55.140 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:40:55.442 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1101, "optime" : { "t" : 1358376057000, "i" : 1297 }, "optimeDate" : ISODate("2013-01-16T22:40:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1090, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1090, "optime" : { "t" : 1358376056000, "i" : 2825 }, "optimeDate" : ISODate("2013-01-16T22:40:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1100, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1088, "optime" : { "t" : 1358376056000, "i" : 49 }, "optimeDate" : ISODate("2013-01-16T22:40:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:40:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1087, "optime" : { "t" : 1358376056000, "i" : 5741 }, "optimeDate" : ISODate("2013-01-16T22:40:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1087, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:40:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1099, "optime" : { "t" : 1358376057000, "i" : 609 }, "optimeDate" : ISODate("2013-01-16T22:40:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:40:57.442 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:40:59.443 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:00.140 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1105, "optime" : { "t" : 1358376061000, "i" : 2569 }, "optimeDate" : ISODate("2013-01-16T22:41:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1094, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1094, "optime" : { "t" : 1358376057000, "i" : 1297 }, "optimeDate" : ISODate("2013-01-16T22:40:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1104, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1092, "optime" : { "t" : 1358376057000, "i" : 1297 }, "optimeDate" : ISODate("2013-01-16T22:40:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1091, "optime" : { "t" : 1358376060000, "i" : 4666 }, "optimeDate" : ISODate("2013-01-16T22:41:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1091, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1103, "optime" : { "t" : 1358376061000, "i" : 2487 }, "optimeDate" : ISODate("2013-01-16T22:41:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:01.388 [conn68] end connection 127.0.0.2:32864 (3 connections now open) m31001| Wed Jan 16 17:41:01.443 [conn67] end connection 127.0.0.2:38518 (2 connections now open) m31001| Wed Jan 16 17:41:01.443 [initandlisten] connection accepted from 127.0.0.2:32801 #68 (3 connections now open) m31000| Wed Jan 16 17:41:01.443 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:01.888 [initandlisten] connection accepted from 127.0.0.2:48052 #70 (4 connections now open) m31000| Wed Jan 16 17:41:03.444 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:04.683 [conn47] end connection 127.0.0.4:47780 (4 connections now open) m31000| Wed Jan 16 17:41:04.683 [initandlisten] connection accepted from 127.0.0.4:58574 #48 (5 connections now open) m31001| Wed Jan 16 17:41:05.141 [conn66] end connection 127.0.0.4:39190 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1109, "optime" : { "t" : 1358376065000, "i" : 3389 }, "optimeDate" : ISODate("2013-01-16T22:41:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1098, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1098, "optime" : { "t" : 1358376064000, "i" : 159 }, "optimeDate" : ISODate("2013-01-16T22:41:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1108, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1096, "optime" : { "t" : 1358376061000, "i" : 2569 }, "optimeDate" : ISODate("2013-01-16T22:41:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1095, "optime" : { "t" : 1358376064000, "i" : 3265 }, "optimeDate" : ISODate("2013-01-16T22:41:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1095, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:04Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1107, "optime" : { "t" : 1358376065000, "i" : 3296 }, "optimeDate" : ISODate("2013-01-16T22:41:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:05.444 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:41:05.641 [initandlisten] connection accepted from 127.0.0.4:45969 #69 (3 connections now open) m31002| Wed Jan 16 17:41:05.641 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:07.444 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:09.445 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1113, "optime" : { "t" : 1358376069000, "i" : 4779 }, "optimeDate" : ISODate("2013-01-16T22:41:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1102, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1102, "optime" : { "t" : 1358376069000, "i" : 3827 }, "optimeDate" : ISODate("2013-01-16T22:41:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1112, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1100, "optime" : { "t" : 1358376065000, "i" : 3389 }, "optimeDate" : ISODate("2013-01-16T22:41:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1099, "optime" : { "t" : 1358376068000, "i" : 2675 }, "optimeDate" : ISODate("2013-01-16T22:41:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1099, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1111, "optime" : { "t" : 1358376069000, "i" : 4297 }, "optimeDate" : ISODate("2013-01-16T22:41:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:10.642 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:11.445 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:41:12.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:41:12.895 [journal] old journal file will be removed: /data/db/testReplSet-0/journal/j._0 m31000| Wed Jan 16 17:41:12.972 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:515032 257ms m31000| Wed Jan 16 17:41:13.445 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1117, "optime" : { "t" : 1358376073000, "i" : 7556 }, "optimeDate" : ISODate("2013-01-16T22:41:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1106, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1106, "optime" : { "t" : 1358376069000, "i" : 4779 }, "optimeDate" : ISODate("2013-01-16T22:41:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1116, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1104, "optime" : { "t" : 1358376069000, "i" : 4779 }, "optimeDate" : ISODate("2013-01-16T22:41:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1103, "optime" : { "t" : 1358376072000, "i" : 1590 }, "optimeDate" : ISODate("2013-01-16T22:41:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1103, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1115, "optime" : { "t" : 1358376073000, "i" : 6905 }, "optimeDate" : ISODate("2013-01-16T22:41:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:15.446 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:15.643 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:17.446 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1121, "optime" : { "t" : 1358376077000, "i" : 5789 }, "optimeDate" : ISODate("2013-01-16T22:41:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1110, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1110, "optime" : { "t" : 1358376076000, "i" : 2898 }, "optimeDate" : ISODate("2013-01-16T22:41:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1120, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1108, "optime" : { "t" : 1358376073000, "i" : 7556 }, "optimeDate" : ISODate("2013-01-16T22:41:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1107, "optime" : { "t" : 1358376076000, "i" : 1367 }, "optimeDate" : ISODate("2013-01-16T22:41:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1107, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1119, "optime" : { "t" : 1358376077000, "i" : 5775 }, "optimeDate" : ISODate("2013-01-16T22:41:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:18.051 [conn69] end connection 127.0.0.3:49436 (3 connections now open) m31002| Wed Jan 16 17:41:18.051 [initandlisten] connection accepted from 127.0.0.3:44936 #71 (4 connections now open) m31000| Wed Jan 16 17:41:19.446 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:20.643 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:21.447 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1125, "optime" : { "t" : 1358376081000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:41:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1114, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1114, "optime" : { "t" : 1358376077000, "i" : 5789 }, "optimeDate" : ISODate("2013-01-16T22:41:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1124, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1112, "optime" : { "t" : 1358376077000, "i" : 5789 }, "optimeDate" : ISODate("2013-01-16T22:41:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1111, "optime" : { "t" : 1358376080000, "i" : 133 }, "optimeDate" : ISODate("2013-01-16T22:41:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1111, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1123, "optime" : { "t" : 1358376081000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:41:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:41:22.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:41:23.447 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:25.447 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:25.644 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1129, "optime" : { "t" : 1358376085000, "i" : 8228 }, "optimeDate" : ISODate("2013-01-16T22:41:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1118, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1118, "optime" : { "t" : 1358376081000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:41:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1129, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1117, "optime" : { "t" : 1358376081000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:41:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1116, "optime" : { "t" : 1358376081000, "i" : 7019 }, "optimeDate" : ISODate("2013-01-16T22:41:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1116, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1128, "optime" : { "t" : 1358376085000, "i" : 8228 }, "optimeDate" : ISODate("2013-01-16T22:41:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:27.448 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:29.448 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:30.022 [journal] old journal file will be removed: /data/db/testReplSet-2/journal/j._0 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1134, "optime" : { "t" : 1358376090000, "i" : 788 }, "optimeDate" : ISODate("2013-01-16T22:41:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1123, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1123, "optime" : { "t" : 1358376089000, "i" : 3328 }, "optimeDate" : ISODate("2013-01-16T22:41:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:40:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1133, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1121, "optime" : { "t" : 1358376089000, "i" : 8174 }, "optimeDate" : ISODate("2013-01-16T22:41:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1120, "optime" : { "t" : 1358376085000, "i" : 8228 }, "optimeDate" : ISODate("2013-01-16T22:41:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1120, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1132, "optime" : { "t" : 1358376089000, "i" : 8176 }, "optimeDate" : ISODate("2013-01-16T22:41:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:30.645 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:41:31.448 [conn68] end connection 127.0.0.2:32801 (2 connections now open) m31001| Wed Jan 16 17:41:31.448 [initandlisten] connection accepted from 127.0.0.2:60569 #70 (3 connections now open) m31000| Wed Jan 16 17:41:31.448 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:33.449 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1138, "optime" : { "t" : 1358376094000, "i" : 2501 }, "optimeDate" : ISODate("2013-01-16T22:41:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1127, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1127, "optime" : { "t" : 1358376090000, "i" : 788 }, "optimeDate" : ISODate("2013-01-16T22:41:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1137, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1125, "optime" : { "t" : 1358376094000, "i" : 249 }, "optimeDate" : ISODate("2013-01-16T22:41:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1124, "optime" : { "t" : 1358376090000, "i" : 788 }, "optimeDate" : ISODate("2013-01-16T22:41:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1124, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1136, "optime" : { "t" : 1358376094000, "i" : 1810 }, "optimeDate" : ISODate("2013-01-16T22:41:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:34.688 [conn48] end connection 127.0.0.4:58574 (4 connections now open) m31000| Wed Jan 16 17:41:34.689 [initandlisten] connection accepted from 127.0.0.4:45439 #49 (5 connections now open) m31000| Wed Jan 16 17:41:35.449 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:35.645 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:37.449 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:41:38.340 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:252918 126ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1142, "optime" : { "t" : 1358376098000, "i" : 3276 }, "optimeDate" : ISODate("2013-01-16T22:41:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1131, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1131, "optime" : { "t" : 1358376094000, "i" : 2501 }, "optimeDate" : ISODate("2013-01-16T22:41:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1141, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1129, "optime" : { "t" : 1358376098000, "i" : 323 }, "optimeDate" : ISODate("2013-01-16T22:41:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1128, "optime" : { "t" : 1358376094000, "i" : 2501 }, "optimeDate" : ISODate("2013-01-16T22:41:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1128, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1140, "optime" : { "t" : 1358376098000, "i" : 2977 }, "optimeDate" : ISODate("2013-01-16T22:41:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:39.393 [conn70] end connection 127.0.0.2:48052 (3 connections now open) m31000| Wed Jan 16 17:41:39.450 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:39.894 [initandlisten] connection accepted from 127.0.0.2:58336 #72 (4 connections now open) m31002| Wed Jan 16 17:41:40.646 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:41.450 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1146, "optime" : { "t" : 1358376102000, "i" : 4387 }, "optimeDate" : ISODate("2013-01-16T22:41:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1135, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1135, "optime" : { "t" : 1358376102000, "i" : 2659 }, "optimeDate" : ISODate("2013-01-16T22:41:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1145, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1133, "optime" : { "t" : 1358376102000, "i" : 130 }, "optimeDate" : ISODate("2013-01-16T22:41:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1132, "optime" : { "t" : 1358376098000, "i" : 3276 }, "optimeDate" : ISODate("2013-01-16T22:41:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1132, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1144, "optime" : { "t" : 1358376102000, "i" : 3975 }, "optimeDate" : ISODate("2013-01-16T22:41:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:41:43.146 [conn69] end connection 127.0.0.4:45969 (2 connections now open) m31000| Wed Jan 16 17:41:43.450 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:41:43.646 [initandlisten] connection accepted from 127.0.0.4:47201 #71 (3 connections now open) m31001| Wed Jan 16 17:41:44.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:41:45.451 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:46.147 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1150, "optime" : { "t" : 1358376106000, "i" : 5748 }, "optimeDate" : ISODate("2013-01-16T22:41:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1139, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1139, "optime" : { "t" : 1358376102000, "i" : 4387 }, "optimeDate" : ISODate("2013-01-16T22:41:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1149, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1137, "optime" : { "t" : 1358376106000, "i" : 203 }, "optimeDate" : ISODate("2013-01-16T22:41:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1136, "optime" : { "t" : 1358376106000, "i" : 5611 }, "optimeDate" : ISODate("2013-01-16T22:41:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1136, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1148, "optime" : { "t" : 1358376106000, "i" : 5748 }, "optimeDate" : ISODate("2013-01-16T22:41:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:47.451 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:48.057 [conn71] end connection 127.0.0.3:44936 (3 connections now open) m31002| Wed Jan 16 17:41:48.057 [initandlisten] connection accepted from 127.0.0.3:38046 #73 (4 connections now open) m31000| Wed Jan 16 17:41:49.452 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1154, "optime" : { "t" : 1358376110000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-16T22:41:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1143, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1143, "optime" : { "t" : 1358376109000, "i" : 1268 }, "optimeDate" : ISODate("2013-01-16T22:41:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1153, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1141, "optime" : { "t" : 1358376110000, "i" : 449 }, "optimeDate" : ISODate("2013-01-16T22:41:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1140, "optime" : { "t" : 1358376110000, "i" : 6210 }, "optimeDate" : ISODate("2013-01-16T22:41:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1140, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1152, "optime" : { "t" : 1358376110000, "i" : 7039 }, "optimeDate" : ISODate("2013-01-16T22:41:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:41:51.148 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:51.452 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:41:53.452 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:41:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1158, "optime" : { "t" : 1358376114000, "i" : 9184 }, "optimeDate" : ISODate("2013-01-16T22:41:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1147, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1147, "optime" : { "t" : 1358376114000, "i" : 8307 }, "optimeDate" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1157, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1145, "optime" : { "t" : 1358376113000, "i" : 816 }, "optimeDate" : ISODate("2013-01-16T22:41:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1144, "optime" : { "t" : 1358376114000, "i" : 6897 }, "optimeDate" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1144, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1156, "optime" : { "t" : 1358376114000, "i" : 8307 }, "optimeDate" : ISODate("2013-01-16T22:41:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:55.453 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:41:56.148 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:41:57.453 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1162, "optime" : { "t" : 1358376118000, "i" : 9760 }, "optimeDate" : ISODate("2013-01-16T22:41:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1151, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1151, "optime" : { "t" : 1358376114000, "i" : 9184 }, "optimeDate" : ISODate("2013-01-16T22:41:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1161, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1149, "optime" : { "t" : 1358376118000, "i" : 1053 }, "optimeDate" : ISODate("2013-01-16T22:41:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:41:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:41:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1148, "optime" : { "t" : 1358376118000, "i" : 7016 }, "optimeDate" : ISODate("2013-01-16T22:41:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1148, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1160, "optime" : { "t" : 1358376118000, "i" : 9667 }, "optimeDate" : ISODate("2013-01-16T22:41:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:41:59.454 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:01.149 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:42:01.454 [conn70] end connection 127.0.0.2:60569 (2 connections now open) m31001| Wed Jan 16 17:42:01.454 [initandlisten] connection accepted from 127.0.0.2:49006 #72 (3 connections now open) m31000| Wed Jan 16 17:42:01.454 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:42:03.113 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:651504 325ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1167, "optime" : { "t" : 1358376123000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:42:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1156, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1156, "optime" : { "t" : 1358376122000, "i" : 3666 }, "optimeDate" : ISODate("2013-01-16T22:42:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1166, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1154, "optime" : { "t" : 1358376122000, "i" : 42 }, "optimeDate" : ISODate("2013-01-16T22:42:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1153, "optime" : { "t" : 1358376122000, "i" : 6092 }, "optimeDate" : ISODate("2013-01-16T22:42:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1153, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1165, "optime" : { "t" : 1358376123000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:42:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:03.455 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:42:04.694 [conn49] end connection 127.0.0.4:45439 (4 connections now open) m31000| Wed Jan 16 17:42:04.694 [initandlisten] connection accepted from 127.0.0.4:58185 #50 (5 connections now open) m31001| Wed Jan 16 17:42:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:42:05.455 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:06.150 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1171, "optime" : { "t" : 1358376127000, "i" : 3342 }, "optimeDate" : ISODate("2013-01-16T22:42:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1160, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1160, "optime" : { "t" : 1358376127000, "i" : 3084 }, "optimeDate" : ISODate("2013-01-16T22:42:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1170, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1158, "optime" : { "t" : 1358376123000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-16T22:42:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 m31000| Wed Jan 16 17:42:07.455 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1157, "optime" : { "t" : 1358376126000, "i" : 4092 }, "optimeDate" : ISODate("2013-01-16T22:42:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1157, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1169, "optime" : { "t" : 1358376127000, "i" : 3084 }, "optimeDate" : ISODate("2013-01-16T22:42:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:09.456 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:11.151 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:42:11.456 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1175, "optime" : { "t" : 1358376131000, "i" : 5583 }, "optimeDate" : ISODate("2013-01-16T22:42:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1164, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1164, "optime" : { "t" : 1358376127000, "i" : 3342 }, "optimeDate" : ISODate("2013-01-16T22:42:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:41:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1174, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1162, "optime" : { "t" : 1358376127000, "i" : 3342 }, "optimeDate" : ISODate("2013-01-16T22:42:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1161, "optime" : { "t" : 1358376130000, "i" : 1705 }, "optimeDate" : ISODate("2013-01-16T22:42:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1161, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1173, "optime" : { "t" : 1358376131000, "i" : 5583 }, "optimeDate" : ISODate("2013-01-16T22:42:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:13.457 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:42:15.457 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1179, "optime" : { "t" : 1358376135000, "i" : 7328 }, "optimeDate" : ISODate("2013-01-16T22:42:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1168, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1168, "optime" : { "t" : 1358376134000, "i" : 1607 }, "optimeDate" : ISODate("2013-01-16T22:42:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1178, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1166, "optime" : { "t" : 1358376131000, "i" : 5583 }, "optimeDate" : ISODate("2013-01-16T22:42:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1165, "optime" : { "t" : 1358376131000, "i" : 5583 }, "optimeDate" : ISODate("2013-01-16T22:42:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1165, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1177, "optime" : { "t" : 1358376135000, "i" : 6491 }, "optimeDate" : ISODate("2013-01-16T22:42:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:42:16.151 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:42:17.398 [conn72] end connection 127.0.0.2:58336 (3 connections now open) m31000| Wed Jan 16 17:42:17.457 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:17.899 [initandlisten] connection accepted from 127.0.0.2:48956 #74 (4 connections now open) m31002| Wed Jan 16 17:42:18.062 [conn73] end connection 127.0.0.3:38046 (3 connections now open) m31002| Wed Jan 16 17:42:18.062 [initandlisten] connection accepted from 127.0.0.3:36851 #75 (4 connections now open) m31000| Wed Jan 16 17:42:19.458 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1183, "optime" : { "t" : 1358376139000, "i" : 8803 }, "optimeDate" : ISODate("2013-01-16T22:42:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1172, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1172, "optime" : { "t" : 1358376135000, "i" : 7328 }, "optimeDate" : ISODate("2013-01-16T22:42:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1182, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1170, "optime" : { "t" : 1358376135000, "i" : 7328 }, "optimeDate" : ISODate("2013-01-16T22:42:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1169, "optime" : { "t" : 1358376135000, "i" : 7328 }, "optimeDate" : ISODate("2013-01-16T22:42:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1169, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1181, "optime" : { "t" : 1358376139000, "i" : 8803 }, "optimeDate" : ISODate("2013-01-16T22:42:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:42:21.152 [conn71] end connection 127.0.0.4:47201 (2 connections now open) m31000| Wed Jan 16 17:42:21.458 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:42:21.652 [initandlisten] connection accepted from 127.0.0.4:52865 #73 (3 connections now open) m31002| Wed Jan 16 17:42:21.652 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:42:23.458 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1188, "optime" : { "t" : 1358376144000, "i" : 748 }, "optimeDate" : ISODate("2013-01-16T22:42:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1177, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1177, "optime" : { "t" : 1358376139000, "i" : 8803 }, "optimeDate" : ISODate("2013-01-16T22:42:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1187, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1175, "optime" : { "t" : 1358376144000, "i" : 91 }, "optimeDate" : ISODate("2013-01-16T22:42:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1174, "optime" : { "t" : 1358376139000, "i" : 8803 }, "optimeDate" : ISODate("2013-01-16T22:42:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:22Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1174, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1186, "optime" : { "t" : 1358376144000, "i" : 91 }, "optimeDate" : ISODate("2013-01-16T22:42:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:25.459 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:26.653 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:42:26.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:42:27.459 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1192, "optime" : { "t" : 1358376148000, "i" : 2584 }, "optimeDate" : ISODate("2013-01-16T22:42:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1181, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1181, "optime" : { "t" : 1358376147000, "i" : 5828 }, "optimeDate" : ISODate("2013-01-16T22:42:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1191, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1179, "optime" : { "t" : 1358376147000, "i" : 5828 }, "optimeDate" : ISODate("2013-01-16T22:42:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1178, "optime" : { "t" : 1358376144000, "i" : 748 }, "optimeDate" : ISODate("2013-01-16T22:42:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1178, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1190, "optime" : { "t" : 1358376148000, "i" : 2254 }, "optimeDate" : ISODate("2013-01-16T22:42:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:29.459 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:42:31.460 [conn72] end connection 127.0.0.2:49006 (2 connections now open) m31001| Wed Jan 16 17:42:31.460 [initandlisten] connection accepted from 127.0.0.2:44379 #74 (3 connections now open) m31000| Wed Jan 16 17:42:31.460 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:31.654 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1196, "optime" : { "t" : 1358376152000, "i" : 3257 }, "optimeDate" : ISODate("2013-01-16T22:42:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1185, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1185, "optime" : { "t" : 1358376148000, "i" : 2584 }, "optimeDate" : ISODate("2013-01-16T22:42:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1195, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1183, "optime" : { "t" : 1358376152000, "i" : 354 }, "optimeDate" : ISODate("2013-01-16T22:42:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1182, "optime" : { "t" : 1358376148000, "i" : 2584 }, "optimeDate" : ISODate("2013-01-16T22:42:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1182, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1194, "optime" : { "t" : 1358376152000, "i" : 3257 }, "optimeDate" : ISODate("2013-01-16T22:42:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:33.460 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:42:34.699 [conn50] end connection 127.0.0.4:58185 (4 connections now open) m31000| Wed Jan 16 17:42:34.700 [initandlisten] connection accepted from 127.0.0.4:43347 #51 (5 connections now open) m31000| Wed Jan 16 17:42:35.461 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1200, "optime" : { "t" : 1358376156000, "i" : 4658 }, "optimeDate" : ISODate("2013-01-16T22:42:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1189, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1189, "optime" : { "t" : 1358376155000, "i" : 34 }, "optimeDate" : ISODate("2013-01-16T22:42:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1199, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1187, "optime" : { "t" : 1358376156000, "i" : 293 }, "optimeDate" : ISODate("2013-01-16T22:42:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1186, "optime" : { "t" : 1358376152000, "i" : 3257 }, "optimeDate" : ISODate("2013-01-16T22:42:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1186, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1198, "optime" : { "t" : 1358376156000, "i" : 4485 }, "optimeDate" : ISODate("2013-01-16T22:42:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:42:36.654 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:42:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:42:37.461 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:42:39.461 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1204, "optime" : { "t" : 1358376160000, "i" : 6487 }, "optimeDate" : ISODate("2013-01-16T22:42:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1193, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1193, "optime" : { "t" : 1358376160000, "i" : 4344 }, "optimeDate" : ISODate("2013-01-16T22:42:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1203, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1191, "optime" : { "t" : 1358376160000, "i" : 1458 }, "optimeDate" : ISODate("2013-01-16T22:42:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1190, "optime" : { "t" : 1358376156000, "i" : 4658 }, "optimeDate" : ISODate("2013-01-16T22:42:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1190, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1202, "optime" : { "t" : 1358376160000, "i" : 6165 }, "optimeDate" : ISODate("2013-01-16T22:42:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:41.462 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:41.655 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:42:43.462 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1208, "optime" : { "t" : 1358376164000, "i" : 6874 }, "optimeDate" : ISODate("2013-01-16T22:42:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1197, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1197, "optime" : { "t" : 1358376160000, "i" : 6487 }, "optimeDate" : ISODate("2013-01-16T22:42:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1207, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1195, "optime" : { "t" : 1358376164000, "i" : 180 }, "optimeDate" : ISODate("2013-01-16T22:42:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1194, "optime" : { "t" : 1358376164000, "i" : 6334 }, "optimeDate" : ISODate("2013-01-16T22:42:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1194, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1206, "optime" : { "t" : 1358376164000, "i" : 6522 }, "optimeDate" : ISODate("2013-01-16T22:42:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:45.463 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:46.656 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:42:47.463 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:48.067 [conn75] end connection 127.0.0.3:36851 (3 connections now open) m31002| Wed Jan 16 17:42:48.067 [initandlisten] connection accepted from 127.0.0.3:47064 #76 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1212, "optime" : { "t" : 1358376168000, "i" : 7871 }, "optimeDate" : ISODate("2013-01-16T22:42:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1201, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1201, "optime" : { "t" : 1358376167000, "i" : 1367 }, "optimeDate" : ISODate("2013-01-16T22:42:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1211, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1199, "optime" : { "t" : 1358376168000, "i" : 811 }, "optimeDate" : ISODate("2013-01-16T22:42:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1198, "optime" : { "t" : 1358376168000, "i" : 6113 }, "optimeDate" : ISODate("2013-01-16T22:42:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1198, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1210, "optime" : { "t" : 1358376168000, "i" : 7378 }, "optimeDate" : ISODate("2013-01-16T22:42:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:49.463 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:42:51.464 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:51.656 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:42:52.945 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:243581 121ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1216, "optime" : { "t" : 1358376172000, "i" : 9279 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1205, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1205, "optime" : { "t" : 1358376172000, "i" : 6944 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1215, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1203, "optime" : { "t" : 1358376172000, "i" : 174 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1202, "optime" : { "t" : 1358376172000, "i" : 6257 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1202, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1214, "optime" : { "t" : 1358376172000, "i" : 9279 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:53.464 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:55.404 [conn74] end connection 127.0.0.2:48956 (3 connections now open) m31000| Wed Jan 16 17:42:55.465 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:42:55.904 [initandlisten] connection accepted from 127.0.0.2:44246 #77 (4 connections now open) m31002| Wed Jan 16 17:42:56.657 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1221, "optime" : { "t" : 1358376177000, "i" : 1632 }, "optimeDate" : ISODate("2013-01-16T22:42:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1210, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1210, "optime" : { "t" : 1358376172000, "i" : 9279 }, "optimeDate" : ISODate("2013-01-16T22:42:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1220, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1208, "optime" : { "t" : 1358376176000, "i" : 299 }, "optimeDate" : ISODate("2013-01-16T22:42:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:42:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1207, "optime" : { "t" : 1358376176000, "i" : 5945 }, "optimeDate" : ISODate("2013-01-16T22:42:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1207, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1219, "optime" : { "t" : 1358376177000, "i" : 1363 }, "optimeDate" : ISODate("2013-01-16T22:42:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:42:57.465 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:42:58.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:42:59.157 [conn73] end connection 127.0.0.4:52865 (2 connections now open) m31000| Wed Jan 16 17:42:59.466 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:42:59.657 [initandlisten] connection accepted from 127.0.0.4:47746 #75 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1225, "optime" : { "t" : 1358376181000, "i" : 2714 }, "optimeDate" : ISODate("2013-01-16T22:43:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1214, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1214, "optime" : { "t" : 1358376180000, "i" : 5793 }, "optimeDate" : ISODate("2013-01-16T22:43:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:42:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1224, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1212, "optime" : { "t" : 1358376177000, "i" : 1632 }, "optimeDate" : ISODate("2013-01-16T22:42:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1211, "optime" : { "t" : 1358376180000, "i" : 4165 }, "optimeDate" : ISODate("2013-01-16T22:43:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1211, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1223, "optime" : { "t" : 1358376181000, "i" : 2714 }, "optimeDate" : ISODate("2013-01-16T22:43:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:43:01.466 [conn74] end connection 127.0.0.2:44379 (2 connections now open) m31001| Wed Jan 16 17:43:01.466 [initandlisten] connection accepted from 127.0.0.2:41469 #76 (3 connections now open) m31000| Wed Jan 16 17:43:01.466 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:02.158 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:03.467 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:04.704 [conn51] end connection 127.0.0.4:43347 (4 connections now open) m31000| Wed Jan 16 17:43:04.705 [initandlisten] connection accepted from 127.0.0.4:40789 #52 (5 connections now open) m31000| Wed Jan 16 17:43:05.467 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1229, "optime" : { "t" : 1358376185000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:43:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1218, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1218, "optime" : { "t" : 1358376181000, "i" : 2714 }, "optimeDate" : ISODate("2013-01-16T22:43:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1228, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1216, "optime" : { "t" : 1358376181000, "i" : 2714 }, "optimeDate" : ISODate("2013-01-16T22:43:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1215, "optime" : { "t" : 1358376184000, "i" : 2876 }, "optimeDate" : ISODate("2013-01-16T22:43:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1215, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1227, "optime" : { "t" : 1358376185000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:43:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:43:07.159 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:07.467 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:43:08.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:43:09.468 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1233, "optime" : { "t" : 1358376189000, "i" : 6239 }, "optimeDate" : ISODate("2013-01-16T22:43:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1222, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1222, "optime" : { "t" : 1358376185000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:43:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1232, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1220, "optime" : { "t" : 1358376185000, "i" : 4406 }, "optimeDate" : ISODate("2013-01-16T22:43:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1219, "optime" : { "t" : 1358376188000, "i" : 1084 }, "optimeDate" : ISODate("2013-01-16T22:43:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1219, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1231, "optime" : { "t" : 1358376189000, "i" : 6239 }, "optimeDate" : ISODate("2013-01-16T22:43:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:11.468 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:12.159 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:12.970 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:439275 219ms m31000| Wed Jan 16 17:43:13.468 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1237, "optime" : { "t" : 1358376193000, "i" : 8637 }, "optimeDate" : ISODate("2013-01-16T22:43:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1226, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1226, "optime" : { "t" : 1358376193000, "i" : 5264 }, "optimeDate" : ISODate("2013-01-16T22:43:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1236, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1224, "optime" : { "t" : 1358376189000, "i" : 6239 }, "optimeDate" : ISODate("2013-01-16T22:43:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:12Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1223, "optime" : { "t" : 1358376189000, "i" : 6239 }, "optimeDate" : ISODate("2013-01-16T22:43:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1223, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1235, "optime" : { "t" : 1358376193000, "i" : 8637 }, "optimeDate" : ISODate("2013-01-16T22:43:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:15.469 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:17.160 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:17.469 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1241, "optime" : { "t" : 1358376197000, "i" : 6826 }, "optimeDate" : ISODate("2013-01-16T22:43:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1230, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1230, "optime" : { "t" : 1358376193000, "i" : 8637 }, "optimeDate" : ISODate("2013-01-16T22:43:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:42:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1240, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1228, "optime" : { "t" : 1358376193000, "i" : 8637 }, "optimeDate" : ISODate("2013-01-16T22:43:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:17Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1227, "optime" : { "t" : 1358376196000, "i" : 136 }, "optimeDate" : ISODate("2013-01-16T22:43:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1227, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1239, "optime" : { "t" : 1358376197000, "i" : 6784 }, "optimeDate" : ISODate("2013-01-16T22:43:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:43:18.073 [conn76] end connection 127.0.0.3:47064 (3 connections now open) m31002| Wed Jan 16 17:43:18.073 [initandlisten] connection accepted from 127.0.0.3:54933 #78 (4 connections now open) m31000| Wed Jan 16 17:43:19.469 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:21.470 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1246, "optime" : { "t" : 1358376202000, "i" : 364 }, "optimeDate" : ISODate("2013-01-16T22:43:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1235, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1235, "optime" : { "t" : 1358376200000, "i" : 298 }, "optimeDate" : ISODate("2013-01-16T22:43:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1245, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1233, "optime" : { "t" : 1358376202000, "i" : 193 }, "optimeDate" : ISODate("2013-01-16T22:43:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1232, "optime" : { "t" : 1358376197000, "i" : 6826 }, "optimeDate" : ISODate("2013-01-16T22:43:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1232, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1244, "optime" : { "t" : 1358376202000, "i" : 364 }, "optimeDate" : ISODate("2013-01-16T22:43:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:43:22.161 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:23.470 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:25.470 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1250, "optime" : { "t" : 1358376206000, "i" : 2422 }, "optimeDate" : ISODate("2013-01-16T22:43:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1239, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1239, "optime" : { "t" : 1358376205000, "i" : 6498 }, "optimeDate" : ISODate("2013-01-16T22:43:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1249, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1237, "optime" : { "t" : 1358376206000, "i" : 312 }, "optimeDate" : ISODate("2013-01-16T22:43:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1236, "optime" : { "t" : 1358376202000, "i" : 364 }, "optimeDate" : ISODate("2013-01-16T22:43:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1236, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1248, "optime" : { "t" : 1358376206000, "i" : 2422 }, "optimeDate" : ISODate("2013-01-16T22:43:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:43:27.161 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:27.471 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:29.471 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1254, "optime" : { "t" : 1358376210000, "i" : 3546 }, "optimeDate" : ISODate("2013-01-16T22:43:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1243, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1243, "optime" : { "t" : 1358376206000, "i" : 2422 }, "optimeDate" : ISODate("2013-01-16T22:43:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1253, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1241, "optime" : { "t" : 1358376210000, "i" : 37 }, "optimeDate" : ISODate("2013-01-16T22:43:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1240, "optime" : { "t" : 1358376206000, "i" : 2422 }, "optimeDate" : ISODate("2013-01-16T22:43:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1240, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1252, "optime" : { "t" : 1358376210000, "i" : 2910 }, "optimeDate" : ISODate("2013-01-16T22:43:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:43:30.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:43:31.471 [conn76] end connection 127.0.0.2:41469 (2 connections now open) m31001| Wed Jan 16 17:43:31.471 [initandlisten] connection accepted from 127.0.0.2:44825 #77 (4 connections now open) m31000| Wed Jan 16 17:43:31.472 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:32.162 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:43:33.409 [conn77] end connection 127.0.0.2:44246 (3 connections now open) m31000| Wed Jan 16 17:43:33.472 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:33.909 [initandlisten] connection accepted from 127.0.0.2:38742 #79 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1258, "optime" : { "t" : 1358376214000, "i" : 4648 }, "optimeDate" : ISODate("2013-01-16T22:43:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1247, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1247, "optime" : { "t" : 1358376213000, "i" : 4257 }, "optimeDate" : ISODate("2013-01-16T22:43:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1257, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1245, "optime" : { "t" : 1358376214000, "i" : 374 }, "optimeDate" : ISODate("2013-01-16T22:43:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1244, "optime" : { "t" : 1358376210000, "i" : 3546 }, "optimeDate" : ISODate("2013-01-16T22:43:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1244, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1256, "optime" : { "t" : 1358376214000, "i" : 4648 }, "optimeDate" : ISODate("2013-01-16T22:43:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:34.710 [conn52] end connection 127.0.0.4:40789 (4 connections now open) m31000| Wed Jan 16 17:43:34.710 [initandlisten] connection accepted from 127.0.0.4:35067 #53 (5 connections now open) m31000| Wed Jan 16 17:43:35.472 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:43:37.163 [conn75] end connection 127.0.0.4:47746 (2 connections now open) m31000| Wed Jan 16 17:43:37.473 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:43:37.663 [initandlisten] connection accepted from 127.0.0.4:46774 #78 (3 connections now open) m31002| Wed Jan 16 17:43:37.663 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1262, "optime" : { "t" : 1358376218000, "i" : 6777 }, "optimeDate" : ISODate("2013-01-16T22:43:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1251, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1251, "optime" : { "t" : 1358376214000, "i" : 4648 }, "optimeDate" : ISODate("2013-01-16T22:43:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1261, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1249, "optime" : { "t" : 1358376218000, "i" : 343 }, "optimeDate" : ISODate("2013-01-16T22:43:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1248, "optime" : { "t" : 1358376218000, "i" : 6383 }, "optimeDate" : ISODate("2013-01-16T22:43:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1248, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1260, "optime" : { "t" : 1358376218000, "i" : 6552 }, "optimeDate" : ISODate("2013-01-16T22:43:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:39.473 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:43:40.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:43:41.474 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:42.664 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1266, "optime" : { "t" : 1358376222000, "i" : 8341 }, "optimeDate" : ISODate("2013-01-16T22:43:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1255, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1255, "optime" : { "t" : 1358376218000, "i" : 6777 }, "optimeDate" : ISODate("2013-01-16T22:43:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1265, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1253, "optime" : { "t" : 1358376222000, "i" : 462 }, "optimeDate" : ISODate("2013-01-16T22:43:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1252, "optime" : { "t" : 1358376222000, "i" : 6092 }, "optimeDate" : ISODate("2013-01-16T22:43:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1252, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1264, "optime" : { "t" : 1358376222000, "i" : 7806 }, "optimeDate" : ISODate("2013-01-16T22:43:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:43.474 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:45.474 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1271, "optime" : { "t" : 1358376227000, "i" : 876 }, "optimeDate" : ISODate("2013-01-16T22:43:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1260, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1260, "optime" : { "t" : 1358376226000, "i" : 3920 }, "optimeDate" : ISODate("2013-01-16T22:43:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1270, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1258, "optime" : { "t" : 1358376226000, "i" : 1076 }, "optimeDate" : ISODate("2013-01-16T22:43:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1257, "optime" : { "t" : 1358376226000, "i" : 6508 }, "optimeDate" : ISODate("2013-01-16T22:43:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1257, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1269, "optime" : { "t" : 1358376227000, "i" : 876 }, "optimeDate" : ISODate("2013-01-16T22:43:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:47.475 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:47.664 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:43:48.078 [conn78] end connection 127.0.0.3:54933 (3 connections now open) m31002| Wed Jan 16 17:43:48.078 [initandlisten] connection accepted from 127.0.0.3:40419 #80 (4 connections now open) m31000| Wed Jan 16 17:43:49.475 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1275, "optime" : { "t" : 1358376231000, "i" : 1993 }, "optimeDate" : ISODate("2013-01-16T22:43:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1264, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1264, "optime" : { "t" : 1358376227000, "i" : 876 }, "optimeDate" : ISODate("2013-01-16T22:43:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1274, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1262, "optime" : { "t" : 1358376227000, "i" : 876 }, "optimeDate" : ISODate("2013-01-16T22:43:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1261, "optime" : { "t" : 1358376230000, "i" : 5248 }, "optimeDate" : ISODate("2013-01-16T22:43:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1261, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1273, "optime" : { "t" : 1358376231000, "i" : 1993 }, "optimeDate" : ISODate("2013-01-16T22:43:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:51.476 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:52.665 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:43:53.476 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1279, "optime" : { "t" : 1358376235000, "i" : 3656 }, "optimeDate" : ISODate("2013-01-16T22:43:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1268, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1268, "optime" : { "t" : 1358376231000, "i" : 1993 }, "optimeDate" : ISODate("2013-01-16T22:43:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1278, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1266, "optime" : { "t" : 1358376231000, "i" : 1993 }, "optimeDate" : ISODate("2013-01-16T22:43:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1265, "optime" : { "t" : 1358376234000, "i" : 3820 }, "optimeDate" : ISODate("2013-01-16T22:43:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1265, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1277, "optime" : { "t" : 1358376235000, "i" : 3656 }, "optimeDate" : ISODate("2013-01-16T22:43:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:55.476 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:43:57.477 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:43:57.666 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1283, "optime" : { "t" : 1358376239000, "i" : 4539 }, "optimeDate" : ISODate("2013-01-16T22:43:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1272, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1272, "optime" : { "t" : 1358376238000, "i" : 4431 }, "optimeDate" : ISODate("2013-01-16T22:43:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1282, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1270, "optime" : { "t" : 1358376235000, "i" : 3656 }, "optimeDate" : ISODate("2013-01-16T22:43:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:43:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1269, "optime" : { "t" : 1358376238000, "i" : 2806 }, "optimeDate" : ISODate("2013-01-16T22:43:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1269, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:43:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1281, "optime" : { "t" : 1358376239000, "i" : 4539 }, "optimeDate" : ISODate("2013-01-16T22:43:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:43:59.477 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:01.477 [conn77] end connection 127.0.0.2:44825 (2 connections now open) m31001| Wed Jan 16 17:44:01.477 [initandlisten] connection accepted from 127.0.0.2:43246 #79 (3 connections now open) m31000| Wed Jan 16 17:44:01.478 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:02.666 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:44:02.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:44:03.004 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:260273 130ms m31000| Wed Jan 16 17:44:03.478 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1287, "optime" : { "t" : 1358376243000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:44:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1276, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1276, "optime" : { "t" : 1358376239000, "i" : 4539 }, "optimeDate" : ISODate("2013-01-16T22:43:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1286, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1274, "optime" : { "t" : 1358376239000, "i" : 4539 }, "optimeDate" : ISODate("2013-01-16T22:43:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1273, "optime" : { "t" : 1358376242000, "i" : 2059 }, "optimeDate" : ISODate("2013-01-16T22:44:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1273, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1285, "optime" : { "t" : 1358376243000, "i" : 5439 }, "optimeDate" : ISODate("2013-01-16T22:44:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:04.715 [conn53] end connection 127.0.0.4:35067 (4 connections now open) m31000| Wed Jan 16 17:44:04.715 [initandlisten] connection accepted from 127.0.0.4:43977 #54 (5 connections now open) m31000| Wed Jan 16 17:44:05.478 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:44:07.479 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1291, "optime" : { "t" : 1358376247000, "i" : 6171 }, "optimeDate" : ISODate("2013-01-16T22:44:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1280, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1280, "optime" : { "t" : 1358376243000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:44:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1290, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1278, "optime" : { "t" : 1358376243000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:44:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1277, "optime" : { "t" : 1358376246000, "i" : 1564 }, "optimeDate" : ISODate("2013-01-16T22:44:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1277, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1289, "optime" : { "t" : 1358376247000, "i" : 6171 }, "optimeDate" : ISODate("2013-01-16T22:44:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:44:07.667 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:09.479 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:11.415 [conn79] end connection 127.0.0.2:38742 (3 connections now open) m31000| Wed Jan 16 17:44:11.479 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1295, "optime" : { "t" : 1358376251000, "i" : 6933 }, "optimeDate" : ISODate("2013-01-16T22:44:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1284, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1284, "optime" : { "t" : 1358376247000, "i" : 6171 }, "optimeDate" : ISODate("2013-01-16T22:44:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1294, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1282, "optime" : { "t" : 1358376247000, "i" : 6171 }, "optimeDate" : ISODate("2013-01-16T22:44:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1281, "optime" : { "t" : 1358376250000, "i" : 672 }, "optimeDate" : ISODate("2013-01-16T22:44:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1281, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1293, "optime" : { "t" : 1358376251000, "i" : 6257 }, "optimeDate" : ISODate("2013-01-16T22:44:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:44:11.915 [initandlisten] connection accepted from 127.0.0.2:36986 #81 (4 connections now open) m31002| Wed Jan 16 17:44:12.668 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:44:12.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:44:13.480 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:15.168 [conn78] end connection 127.0.0.4:46774 (2 connections now open) m31000| Wed Jan 16 17:44:15.480 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:15.668 [initandlisten] connection accepted from 127.0.0.4:43354 #80 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1299, "optime" : { "t" : 1358376255000, "i" : 8219 }, "optimeDate" : ISODate("2013-01-16T22:44:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1288, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1288, "optime" : { "t" : 1358376251000, "i" : 6933 }, "optimeDate" : ISODate("2013-01-16T22:44:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1298, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1286, "optime" : { "t" : 1358376251000, "i" : 6933 }, "optimeDate" : ISODate("2013-01-16T22:44:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:15Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1285, "optime" : { "t" : 1358376251000, "i" : 6933 }, "optimeDate" : ISODate("2013-01-16T22:44:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1285, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1297, "optime" : { "t" : 1358376255000, "i" : 8125 }, "optimeDate" : ISODate("2013-01-16T22:44:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:17.480 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:18.084 [conn80] end connection 127.0.0.3:40419 (3 connections now open) m31002| Wed Jan 16 17:44:18.084 [initandlisten] connection accepted from 127.0.0.3:52357 #82 (4 connections now open) m31002| Wed Jan 16 17:44:18.168 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:19.481 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1303, "optime" : { "t" : 1358376259000, "i" : 9036 }, "optimeDate" : ISODate("2013-01-16T22:44:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1292, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1292, "optime" : { "t" : 1358376259000, "i" : 3563 }, "optimeDate" : ISODate("2013-01-16T22:44:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:43:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1302, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1290, "optime" : { "t" : 1358376255000, "i" : 8219 }, "optimeDate" : ISODate("2013-01-16T22:44:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1289, "optime" : { "t" : 1358376255000, "i" : 8219 }, "optimeDate" : ISODate("2013-01-16T22:44:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1289, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1301, "optime" : { "t" : 1358376259000, "i" : 9036 }, "optimeDate" : ISODate("2013-01-16T22:44:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:21.481 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:23.169 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:23.481 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1308, "optime" : { "t" : 1358376264000, "i" : 936 }, "optimeDate" : ISODate("2013-01-16T22:44:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1297, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1297, "optime" : { "t" : 1358376259000, "i" : 9036 }, "optimeDate" : ISODate("2013-01-16T22:44:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1307, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1295, "optime" : { "t" : 1358376264000, "i" : 554 }, "optimeDate" : ISODate("2013-01-16T22:44:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1294, "optime" : { "t" : 1358376259000, "i" : 9036 }, "optimeDate" : ISODate("2013-01-16T22:44:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1294, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1306, "optime" : { "t" : 1358376264000, "i" : 680 }, "optimeDate" : ISODate("2013-01-16T22:44:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:25.482 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:44:27.482 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:28.170 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1312, "optime" : { "t" : 1358376268000, "i" : 2566 }, "optimeDate" : ISODate("2013-01-16T22:44:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1301, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1301, "optime" : { "t" : 1358376264000, "i" : 936 }, "optimeDate" : ISODate("2013-01-16T22:44:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1311, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1299, "optime" : { "t" : 1358376268000, "i" : 1087 }, "optimeDate" : ISODate("2013-01-16T22:44:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1298, "optime" : { "t" : 1358376264000, "i" : 936 }, "optimeDate" : ISODate("2013-01-16T22:44:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1298, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1310, "optime" : { "t" : 1358376268000, "i" : 2554 }, "optimeDate" : ISODate("2013-01-16T22:44:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:29.482 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:31.483 [conn79] end connection 127.0.0.2:43246 (2 connections now open) m31001| Wed Jan 16 17:44:31.483 [initandlisten] connection accepted from 127.0.0.2:56120 #81 (3 connections now open) m31000| Wed Jan 16 17:44:31.483 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1316, "optime" : { "t" : 1358376272000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:44:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1305, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1305, "optime" : { "t" : 1358376271000, "i" : 4856 }, "optimeDate" : ISODate("2013-01-16T22:44:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1315, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1303, "optime" : { "t" : 1358376272000, "i" : 242 }, "optimeDate" : ISODate("2013-01-16T22:44:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1302, "optime" : { "t" : 1358376268000, "i" : 2566 }, "optimeDate" : ISODate("2013-01-16T22:44:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1302, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1314, "optime" : { "t" : 1358376272000, "i" : 4032 }, "optimeDate" : ISODate("2013-01-16T22:44:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:44:33.170 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:33.483 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:44:34.721 [conn54] end connection 127.0.0.4:43977 (4 connections now open) m31000| Wed Jan 16 17:44:34.721 [initandlisten] connection accepted from 127.0.0.4:52977 #55 (5 connections now open) m31001| Wed Jan 16 17:44:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:44:35.484 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1320, "optime" : { "t" : 1358376276000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:44:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1309, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1309, "optime" : { "t" : 1358376272000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:44:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1319, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1307, "optime" : { "t" : 1358376276000, "i" : 673 }, "optimeDate" : ISODate("2013-01-16T22:44:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1306, "optime" : { "t" : 1358376272000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:44:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1306, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1318, "optime" : { "t" : 1358376276000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:44:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:37.484 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:38.171 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:39.484 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1324, "optime" : { "t" : 1358376280000, "i" : 7061 }, "optimeDate" : ISODate("2013-01-16T22:44:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1313, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1313, "optime" : { "t" : 1358376276000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:44:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1323, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1311, "optime" : { "t" : 1358376280000, "i" : 493 }, "optimeDate" : ISODate("2013-01-16T22:44:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1310, "optime" : { "t" : 1358376280000, "i" : 6398 }, "optimeDate" : ISODate("2013-01-16T22:44:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1310, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1322, "optime" : { "t" : 1358376280000, "i" : 7061 }, "optimeDate" : ISODate("2013-01-16T22:44:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:41.485 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:43.172 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:43.485 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:44.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1328, "optime" : { "t" : 1358376284000, "i" : 8532 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1317, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1317, "optime" : { "t" : 1358376284000, "i" : 3303 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1327, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1315, "optime" : { "t" : 1358376284000, "i" : 46 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1314, "optime" : { "t" : 1358376284000, "i" : 6322 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1314, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1326, "optime" : { "t" : 1358376284000, "i" : 7914 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:45.485 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:44:47.486 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:48.089 [conn82] end connection 127.0.0.3:52357 (3 connections now open) m31002| Wed Jan 16 17:44:48.089 [initandlisten] connection accepted from 127.0.0.3:34267 #83 (5 connections now open) m31002| Wed Jan 16 17:44:48.172 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1333, "optime" : { "t" : 1358376289000, "i" : 749 }, "optimeDate" : ISODate("2013-01-16T22:44:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1322, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1322, "optime" : { "t" : 1358376284000, "i" : 8532 }, "optimeDate" : ISODate("2013-01-16T22:44:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1332, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1320, "optime" : { "t" : 1358376288000, "i" : 1145 }, "optimeDate" : ISODate("2013-01-16T22:44:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1319, "optime" : { "t" : 1358376288000, "i" : 7233 }, "optimeDate" : ISODate("2013-01-16T22:44:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1319, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1331, "optime" : { "t" : 1358376289000, "i" : 749 }, "optimeDate" : ISODate("2013-01-16T22:44:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:44:49.420 [conn81] end connection 127.0.0.2:36986 (3 connections now open) m31000| Wed Jan 16 17:44:49.486 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:49.920 [initandlisten] connection accepted from 127.0.0.2:48496 #84 (4 connections now open) m31000| Wed Jan 16 17:44:51.487 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:53.173 [conn80] end connection 127.0.0.4:43354 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1337, "optime" : { "t" : 1358376293000, "i" : 1422 }, "optimeDate" : ISODate("2013-01-16T22:44:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1326, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1326, "optime" : { "t" : 1358376292000, "i" : 2683 }, "optimeDate" : ISODate("2013-01-16T22:44:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1336, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1324, "optime" : { "t" : 1358376292000, "i" : 144 }, "optimeDate" : ISODate("2013-01-16T22:44:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1323, "optime" : { "t" : 1358376292000, "i" : 5783 }, "optimeDate" : ISODate("2013-01-16T22:44:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1323, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1335, "optime" : { "t" : 1358376293000, "i" : 1183 }, "optimeDate" : ISODate("2013-01-16T22:44:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:53.487 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:44:53.673 [initandlisten] connection accepted from 127.0.0.4:45079 #82 (3 connections now open) m31002| Wed Jan 16 17:44:53.673 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:55.487 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1341, "optime" : { "t" : 1358376297000, "i" : 3136 }, "optimeDate" : ISODate("2013-01-16T22:44:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1330, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1330, "optime" : { "t" : 1358376293000, "i" : 1422 }, "optimeDate" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1340, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1328, "optime" : { "t" : 1358376293000, "i" : 1422 }, "optimeDate" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:44:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1327, "optime" : { "t" : 1358376296000, "i" : 4205 }, "optimeDate" : ISODate("2013-01-16T22:44:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1327, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1339, "optime" : { "t" : 1358376297000, "i" : 3136 }, "optimeDate" : ISODate("2013-01-16T22:44:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:44:57.488 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:44:58.674 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:44:59.488 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1345, "optime" : { "t" : 1358376301000, "i" : 3770 }, "optimeDate" : ISODate("2013-01-16T22:45:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1334, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1334, "optime" : { "t" : 1358376297000, "i" : 3136 }, "optimeDate" : ISODate("2013-01-16T22:44:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:44:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1344, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1332, "optime" : { "t" : 1358376297000, "i" : 3136 }, "optimeDate" : ISODate("2013-01-16T22:44:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:01Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1331, "optime" : { "t" : 1358376300000, "i" : 3252 }, "optimeDate" : ISODate("2013-01-16T22:45:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1331, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1343, "optime" : { "t" : 1358376301000, "i" : 3770 }, "optimeDate" : ISODate("2013-01-16T22:45:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:45:01.489 [conn81] end connection 127.0.0.2:56120 (2 connections now open) m31001| Wed Jan 16 17:45:01.489 [initandlisten] connection accepted from 127.0.0.2:39515 #83 (3 connections now open) m31000| Wed Jan 16 17:45:01.489 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:03.490 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:03.675 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:04.726 [conn55] end connection 127.0.0.4:52977 (4 connections now open) m31000| Wed Jan 16 17:45:04.726 [initandlisten] connection accepted from 127.0.0.4:33853 #56 (6 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:45:05.490 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1349, "optime" : { "t" : 1358376305000, "i" : 5003 }, "optimeDate" : ISODate("2013-01-16T22:45:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1338, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1338, "optime" : { "t" : 1358376304000, "i" : 4421 }, "optimeDate" : ISODate("2013-01-16T22:45:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1348, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1336, "optime" : { "t" : 1358376301000, "i" : 3770 }, "optimeDate" : ISODate("2013-01-16T22:45:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1335, "optime" : { "t" : 1358376304000, "i" : 2653 }, "optimeDate" : ISODate("2013-01-16T22:45:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1335, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:04Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1347, "optime" : { "t" : 1358376305000, "i" : 5003 }, "optimeDate" : ISODate("2013-01-16T22:45:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:45:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:45:07.490 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:08.675 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:09.491 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1353, "optime" : { "t" : 1358376309000, "i" : 6184 }, "optimeDate" : ISODate("2013-01-16T22:45:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1342, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1342, "optime" : { "t" : 1358376305000, "i" : 5003 }, "optimeDate" : ISODate("2013-01-16T22:45:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1352, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1340, "optime" : { "t" : 1358376305000, "i" : 5003 }, "optimeDate" : ISODate("2013-01-16T22:45:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1339, "optime" : { "t" : 1358376308000, "i" : 1585 }, "optimeDate" : ISODate("2013-01-16T22:45:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1339, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1351, "optime" : { "t" : 1358376309000, "i" : 6184 }, "optimeDate" : ISODate("2013-01-16T22:45:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:11.491 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:12.967 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:310456 155ms m31000| Wed Jan 16 17:45:13.491 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:13.676 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1357, "optime" : { "t" : 1358376313000, "i" : 8032 }, "optimeDate" : ISODate("2013-01-16T22:45:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1346, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1346, "optime" : { "t" : 1358376309000, "i" : 6184 }, "optimeDate" : ISODate("2013-01-16T22:45:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1356, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1344, "optime" : { "t" : 1358376309000, "i" : 6184 }, "optimeDate" : ISODate("2013-01-16T22:45:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1343, "optime" : { "t" : 1358376312000, "i" : 6 }, "optimeDate" : ISODate("2013-01-16T22:45:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1343, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1355, "optime" : { "t" : 1358376313000, "i" : 7223 }, "optimeDate" : ISODate("2013-01-16T22:45:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:15.492 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:45:16.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:45:17.492 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:45:17.971 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:447429 223ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1362, "optime" : { "t" : 1358376318000, "i" : 747 }, "optimeDate" : ISODate("2013-01-16T22:45:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1351, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1351, "optime" : { "t" : 1358376317000, "i" : 3969 }, "optimeDate" : ISODate("2013-01-16T22:45:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1361, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1349, "optime" : { "t" : 1358376313000, "i" : 8032 }, "optimeDate" : ISODate("2013-01-16T22:45:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1348, "optime" : { "t" : 1358376313000, "i" : 8032 }, "optimeDate" : ISODate("2013-01-16T22:45:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1348, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1360, "optime" : { "t" : 1358376318000, "i" : 747 }, "optimeDate" : ISODate("2013-01-16T22:45:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:45:18.095 [conn83] end connection 127.0.0.3:34267 (3 connections now open) m31002| Wed Jan 16 17:45:18.095 [initandlisten] connection accepted from 127.0.0.3:47214 #85 (4 connections now open) m31002| Wed Jan 16 17:45:18.676 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:19.492 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:21.493 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1366, "optime" : { "t" : 1358376322000, "i" : 1208 }, "optimeDate" : ISODate("2013-01-16T22:45:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1355, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1355, "optime" : { "t" : 1358376318000, "i" : 747 }, "optimeDate" : ISODate("2013-01-16T22:45:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1365, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1353, "optime" : { "t" : 1358376322000, "i" : 506 }, "optimeDate" : ISODate("2013-01-16T22:45:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1352, "optime" : { "t" : 1358376318000, "i" : 747 }, "optimeDate" : ISODate("2013-01-16T22:45:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1352, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1364, "optime" : { "t" : 1358376322000, "i" : 507 }, "optimeDate" : ISODate("2013-01-16T22:45:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:23.493 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:23.677 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:25.493 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1370, "optime" : { "t" : 1358376326000, "i" : 2919 }, "optimeDate" : ISODate("2013-01-16T22:45:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1359, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1359, "optime" : { "t" : 1358376322000, "i" : 1208 }, "optimeDate" : ISODate("2013-01-16T22:45:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:44:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1369, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1357, "optime" : { "t" : 1358376326000, "i" : 1290 }, "optimeDate" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1356, "optime" : { "t" : 1358376322000, "i" : 1208 }, "optimeDate" : ISODate("2013-01-16T22:45:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1356, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1368, "optime" : { "t" : 1358376326000, "i" : 2919 }, "optimeDate" : ISODate("2013-01-16T22:45:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:45:27.425 [conn84] end connection 127.0.0.2:48496 (3 connections now open) m31000| Wed Jan 16 17:45:27.494 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:27.925 [initandlisten] connection accepted from 127.0.0.2:48138 #86 (4 connections now open) m31002| Wed Jan 16 17:45:28.678 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:29.494 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1374, "optime" : { "t" : 1358376330000, "i" : 3494 }, "optimeDate" : ISODate("2013-01-16T22:45:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1363, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1363, "optime" : { "t" : 1358376326000, "i" : 2919 }, "optimeDate" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1373, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1361, "optime" : { "t" : 1358376330000, "i" : 978 }, "optimeDate" : ISODate("2013-01-16T22:45:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1360, "optime" : { "t" : 1358376326000, "i" : 2919 }, "optimeDate" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1360, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1372, "optime" : { "t" : 1358376330000, "i" : 3494 }, "optimeDate" : ISODate("2013-01-16T22:45:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:45:31.178 [conn82] end connection 127.0.0.4:45079 (2 connections now open) m31001| Wed Jan 16 17:45:31.494 [conn83] end connection 127.0.0.2:39515 (1 connection now open) m31001| Wed Jan 16 17:45:31.494 [initandlisten] connection accepted from 127.0.0.2:58229 #84 (2 connections now open) m31000| Wed Jan 16 17:45:31.495 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:45:31.678 [initandlisten] connection accepted from 127.0.0.4:35747 #85 (3 connections now open) m31000| Wed Jan 16 17:45:33.495 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:34.179 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1378, "optime" : { "t" : 1358376334000, "i" : 4587 }, "optimeDate" : ISODate("2013-01-16T22:45:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1367, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1367, "optime" : { "t" : 1358376330000, "i" : 3494 }, "optimeDate" : ISODate("2013-01-16T22:45:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1377, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1365, "optime" : { "t" : 1358376334000, "i" : 1186 }, "optimeDate" : ISODate("2013-01-16T22:45:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1364, "optime" : { "t" : 1358376330000, "i" : 3494 }, "optimeDate" : ISODate("2013-01-16T22:45:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1364, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1376, "optime" : { "t" : 1358376334000, "i" : 4153 }, "optimeDate" : ISODate("2013-01-16T22:45:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:34.732 [conn56] end connection 127.0.0.4:33853 (4 connections now open) m31000| Wed Jan 16 17:45:34.732 [initandlisten] connection accepted from 127.0.0.4:60610 #57 (5 connections now open) m31000| Wed Jan 16 17:45:35.495 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:37.495 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:38.191 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:233292 116ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1382, "optime" : { "t" : 1358376338000, "i" : 5305 }, "optimeDate" : ISODate("2013-01-16T22:45:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1371, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1371, "optime" : { "t" : 1358376337000, "i" : 3997 }, "optimeDate" : ISODate("2013-01-16T22:45:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1381, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1369, "optime" : { "t" : 1358376338000, "i" : 578 }, "optimeDate" : ISODate("2013-01-16T22:45:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1368, "optime" : { "t" : 1358376334000, "i" : 4587 }, "optimeDate" : ISODate("2013-01-16T22:45:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1368, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1380, "optime" : { "t" : 1358376338000, "i" : 5305 }, "optimeDate" : ISODate("2013-01-16T22:45:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:45:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:45:39.179 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:39.496 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:41.496 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1386, "optime" : { "t" : 1358376342000, "i" : 6788 }, "optimeDate" : ISODate("2013-01-16T22:45:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1375, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1375, "optime" : { "t" : 1358376338000, "i" : 5305 }, "optimeDate" : ISODate("2013-01-16T22:45:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1385, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1373, "optime" : { "t" : 1358376342000, "i" : 1191 }, "optimeDate" : ISODate("2013-01-16T22:45:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1372, "optime" : { "t" : 1358376338000, "i" : 5305 }, "optimeDate" : ISODate("2013-01-16T22:45:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1372, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1384, "optime" : { "t" : 1358376342000, "i" : 6788 }, "optimeDate" : ISODate("2013-01-16T22:45:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:43.497 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:44.180 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:45.497 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1390, "optime" : { "t" : 1358376346000, "i" : 8045 }, "optimeDate" : ISODate("2013-01-16T22:45:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1379, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1379, "optime" : { "t" : 1358376342000, "i" : 6788 }, "optimeDate" : ISODate("2013-01-16T22:45:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1389, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1377, "optime" : { "t" : 1358376346000, "i" : 1522 }, "optimeDate" : ISODate("2013-01-16T22:45:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1376, "optime" : { "t" : 1358376346000, "i" : 7516 }, "optimeDate" : ISODate("2013-01-16T22:45:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1376, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1388, "optime" : { "t" : 1358376346000, "i" : 7714 }, "optimeDate" : ISODate("2013-01-16T22:45:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:47.497 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:48.100 [conn85] end connection 127.0.0.3:47214 (3 connections now open) m31002| Wed Jan 16 17:45:48.100 [initandlisten] connection accepted from 127.0.0.3:36944 #87 (4 connections now open) m31001| Wed Jan 16 17:45:48.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:45:49.181 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:45:49.498 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1395, "optime" : { "t" : 1358376351000, "i" : 380 }, "optimeDate" : ISODate("2013-01-16T22:45:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1384, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1384, "optime" : { "t" : 1358376350000, "i" : 3580 }, "optimeDate" : ISODate("2013-01-16T22:45:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1394, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1382, "optime" : { "t" : 1358376350000, "i" : 311 }, "optimeDate" : ISODate("2013-01-16T22:45:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1381, "optime" : { "t" : 1358376350000, "i" : 6273 }, "optimeDate" : ISODate("2013-01-16T22:45:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1381, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1393, "optime" : { "t" : 1358376351000, "i" : 380 }, "optimeDate" : ISODate("2013-01-16T22:45:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:51.498 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:53.499 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:54.181 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1399, "optime" : { "t" : 1358376355000, "i" : 1714 }, "optimeDate" : ISODate("2013-01-16T22:45:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1388, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1388, "optime" : { "t" : 1358376351000, "i" : 380 }, "optimeDate" : ISODate("2013-01-16T22:45:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1398, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1386, "optime" : { "t" : 1358376354000, "i" : 582 }, "optimeDate" : ISODate("2013-01-16T22:45:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1385, "optime" : { "t" : 1358376354000, "i" : 5803 }, "optimeDate" : ISODate("2013-01-16T22:45:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1385, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1397, "optime" : { "t" : 1358376355000, "i" : 1714 }, "optimeDate" : ISODate("2013-01-16T22:45:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:55.499 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:45:57.499 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:45:59.182 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1403, "optime" : { "t" : 1358376359000, "i" : 3463 }, "optimeDate" : ISODate("2013-01-16T22:45:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1392, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1392, "optime" : { "t" : 1358376355000, "i" : 1714 }, "optimeDate" : ISODate("2013-01-16T22:45:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1402, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1390, "optime" : { "t" : 1358376355000, "i" : 1714 }, "optimeDate" : ISODate("2013-01-16T22:45:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:45:59Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:45:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1389, "optime" : { "t" : 1358376358000, "i" : 4188 }, "optimeDate" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1389, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1401, "optime" : { "t" : 1358376359000, "i" : 3463 }, "optimeDate" : ISODate("2013-01-16T22:45:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:45:59.500 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:01.500 [conn84] end connection 127.0.0.2:58229 (2 connections now open) m31001| Wed Jan 16 17:46:01.500 [initandlisten] connection accepted from 127.0.0.2:55013 #86 (3 connections now open) m31000| Wed Jan 16 17:46:01.501 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1407, "optime" : { "t" : 1358376363000, "i" : 4647 }, "optimeDate" : ISODate("2013-01-16T22:46:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1396, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1396, "optime" : { "t" : 1358376362000, "i" : 5050 }, "optimeDate" : ISODate("2013-01-16T22:46:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1406, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1394, "optime" : { "t" : 1358376359000, "i" : 3463 }, "optimeDate" : ISODate("2013-01-16T22:45:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1393, "optime" : { "t" : 1358376362000, "i" : 3192 }, "optimeDate" : ISODate("2013-01-16T22:46:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1393, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1405, "optime" : { "t" : 1358376363000, "i" : 3850 }, "optimeDate" : ISODate("2013-01-16T22:46:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:03.501 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:04.182 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:04.737 [conn57] end connection 127.0.0.4:60610 (4 connections now open) m31000| Wed Jan 16 17:46:04.737 [initandlisten] connection accepted from 127.0.0.4:34253 #58 (5 connections now open) m31002| Wed Jan 16 17:46:05.430 [conn86] end connection 127.0.0.2:48138 (3 connections now open) m31000| Wed Jan 16 17:46:05.501 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:05.931 [initandlisten] connection accepted from 127.0.0.2:33214 #88 (4 connections now open) m31000| Wed Jan 16 17:46:07.502 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1411, "optime" : { "t" : 1358376367000, "i" : 5386 }, "optimeDate" : ISODate("2013-01-16T22:46:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1400, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1400, "optime" : { "t" : 1358376363000, "i" : 4647 }, "optimeDate" : ISODate("2013-01-16T22:46:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1410, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1398, "optime" : { "t" : 1358376363000, "i" : 4647 }, "optimeDate" : ISODate("2013-01-16T22:46:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1397, "optime" : { "t" : 1358376366000, "i" : 1937 }, "optimeDate" : ISODate("2013-01-16T22:46:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1397, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1409, "optime" : { "t" : 1358376367000, "i" : 4657 }, "optimeDate" : ISODate("2013-01-16T22:46:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:46:09.183 [conn85] end connection 127.0.0.4:35747 (2 connections now open) m31000| Wed Jan 16 17:46:09.502 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:09.683 [initandlisten] connection accepted from 127.0.0.4:42467 #87 (3 connections now open) m31002| Wed Jan 16 17:46:09.683 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:46:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:46:11.502 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1415, "optime" : { "t" : 1358376371000, "i" : 7005 }, "optimeDate" : ISODate("2013-01-16T22:46:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1404, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1404, "optime" : { "t" : 1358376370000, "i" : 2829 }, "optimeDate" : ISODate("2013-01-16T22:46:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1414, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1402, "optime" : { "t" : 1358376367000, "i" : 5386 }, "optimeDate" : ISODate("2013-01-16T22:46:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1401, "optime" : { "t" : 1358376370000, "i" : 1293 }, "optimeDate" : ISODate("2013-01-16T22:46:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1401, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1413, "optime" : { "t" : 1358376371000, "i" : 7005 }, "optimeDate" : ISODate("2013-01-16T22:46:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:13.503 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:14.684 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:15.503 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1419, "optime" : { "t" : 1358376375000, "i" : 8355 }, "optimeDate" : ISODate("2013-01-16T22:46:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1408, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1408, "optime" : { "t" : 1358376371000, "i" : 7005 }, "optimeDate" : ISODate("2013-01-16T22:46:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1418, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1406, "optime" : { "t" : 1358376371000, "i" : 7005 }, "optimeDate" : ISODate("2013-01-16T22:46:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1405, "optime" : { "t" : 1358376371000, "i" : 7005 }, "optimeDate" : ISODate("2013-01-16T22:46:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1405, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1417, "optime" : { "t" : 1358376375000, "i" : 8355 }, "optimeDate" : ISODate("2013-01-16T22:46:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:17.504 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:18.105 [conn87] end connection 127.0.0.3:36944 (3 connections now open) m31002| Wed Jan 16 17:46:18.105 [initandlisten] connection accepted from 127.0.0.3:36514 #89 (4 connections now open) m31000| Wed Jan 16 17:46:19.504 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:19.685 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1423, "optime" : { "t" : 1358376379000, "i" : 9579 }, "optimeDate" : ISODate("2013-01-16T22:46:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1412, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1412, "optime" : { "t" : 1358376375000, "i" : 8355 }, "optimeDate" : ISODate("2013-01-16T22:46:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1422, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1410, "optime" : { "t" : 1358376375000, "i" : 8355 }, "optimeDate" : ISODate("2013-01-16T22:46:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:19Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1409, "optime" : { "t" : 1358376375000, "i" : 8355 }, "optimeDate" : ISODate("2013-01-16T22:46:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1409, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1421, "optime" : { "t" : 1358376379000, "i" : 9088 }, "optimeDate" : ISODate("2013-01-16T22:46:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:46:20.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:46:21.504 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:46:23.505 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1428, "optime" : { "t" : 1358376384000, "i" : 1233 }, "optimeDate" : ISODate("2013-01-16T22:46:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1417, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1417, "optime" : { "t" : 1358376383000, "i" : 3811 }, "optimeDate" : ISODate("2013-01-16T22:46:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1427, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1415, "optime" : { "t" : 1358376384000, "i" : 1080 }, "optimeDate" : ISODate("2013-01-16T22:46:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1414, "optime" : { "t" : 1358376379000, "i" : 9579 }, "optimeDate" : ISODate("2013-01-16T22:46:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1414, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1426, "optime" : { "t" : 1358376384000, "i" : 1233 }, "optimeDate" : ISODate("2013-01-16T22:46:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:46:24.685 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:25.505 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:46:27.505 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:46:27.979 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:277070 138ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1432, "optime" : { "t" : 1358376388000, "i" : 3645 }, "optimeDate" : ISODate("2013-01-16T22:46:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1421, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1421, "optime" : { "t" : 1358376384000, "i" : 1233 }, "optimeDate" : ISODate("2013-01-16T22:46:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:45:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1431, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1419, "optime" : { "t" : 1358376388000, "i" : 1349 }, "optimeDate" : ISODate("2013-01-16T22:46:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1418, "optime" : { "t" : 1358376384000, "i" : 1233 }, "optimeDate" : ISODate("2013-01-16T22:46:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1418, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1430, "optime" : { "t" : 1358376388000, "i" : 3518 }, "optimeDate" : ISODate("2013-01-16T22:46:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:29.506 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:29.686 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:46:31.506 [conn86] end connection 127.0.0.2:55013 (2 connections now open) m31001| Wed Jan 16 17:46:31.506 [initandlisten] connection accepted from 127.0.0.2:37346 #88 (3 connections now open) m31000| Wed Jan 16 17:46:31.506 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1436, "optime" : { "t" : 1358376392000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-16T22:46:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1425, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1425, "optime" : { "t" : 1358376388000, "i" : 3645 }, "optimeDate" : ISODate("2013-01-16T22:46:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1435, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1423, "optime" : { "t" : 1358376392000, "i" : 636 }, "optimeDate" : ISODate("2013-01-16T22:46:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1422, "optime" : { "t" : 1358376388000, "i" : 3645 }, "optimeDate" : ISODate("2013-01-16T22:46:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1422, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1434, "optime" : { "t" : 1358376392000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-16T22:46:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:33.506 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:34.687 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:34.742 [conn58] end connection 127.0.0.4:34253 (4 connections now open) m31000| Wed Jan 16 17:46:34.743 [initandlisten] connection accepted from 127.0.0.4:45611 #59 (5 connections now open) m31000| Wed Jan 16 17:46:35.507 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1440, "optime" : { "t" : 1358376396000, "i" : 4569 }, "optimeDate" : ISODate("2013-01-16T22:46:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1429, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1429, "optime" : { "t" : 1358376395000, "i" : 5125 }, "optimeDate" : ISODate("2013-01-16T22:46:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1439, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1427, "optime" : { "t" : 1358376396000, "i" : 880 }, "optimeDate" : ISODate("2013-01-16T22:46:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1426, "optime" : { "t" : 1358376392000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-16T22:46:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1426, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1438, "optime" : { "t" : 1358376396000, "i" : 4569 }, "optimeDate" : ISODate("2013-01-16T22:46:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:37.507 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:46:39.507 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:39.687 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1444, "optime" : { "t" : 1358376400000, "i" : 6058 }, "optimeDate" : ISODate("2013-01-16T22:46:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1433, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1433, "optime" : { "t" : 1358376396000, "i" : 4569 }, "optimeDate" : ISODate("2013-01-16T22:46:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1443, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1431, "optime" : { "t" : 1358376400000, "i" : 566 }, "optimeDate" : ISODate("2013-01-16T22:46:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1430, "optime" : { "t" : 1358376396000, "i" : 4569 }, "optimeDate" : ISODate("2013-01-16T22:46:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1430, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1442, "optime" : { "t" : 1358376400000, "i" : 6058 }, "optimeDate" : ISODate("2013-01-16T22:46:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:41.508 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:46:43.436 [conn88] end connection 127.0.0.2:33214 (3 connections now open) m31000| Wed Jan 16 17:46:43.508 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:43.936 [initandlisten] connection accepted from 127.0.0.2:35319 #90 (4 connections now open) m31002| Wed Jan 16 17:46:44.688 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1448, "optime" : { "t" : 1358376404000, "i" : 7474 }, "optimeDate" : ISODate("2013-01-16T22:46:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1437, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1437, "optime" : { "t" : 1358376403000, "i" : 1616 }, "optimeDate" : ISODate("2013-01-16T22:46:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1447, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1435, "optime" : { "t" : 1358376404000, "i" : 338 }, "optimeDate" : ISODate("2013-01-16T22:46:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1434, "optime" : { "t" : 1358376404000, "i" : 6538 }, "optimeDate" : ISODate("2013-01-16T22:46:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1434, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1446, "optime" : { "t" : 1358376404000, "i" : 6878 }, "optimeDate" : ISODate("2013-01-16T22:46:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:45.508 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:47.188 [conn87] end connection 127.0.0.4:42467 (2 connections now open) m31000| Wed Jan 16 17:46:47.509 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:47.688 [initandlisten] connection accepted from 127.0.0.4:45104 #89 (3 connections now open) m31002| Wed Jan 16 17:46:48.110 [conn89] end connection 127.0.0.3:36514 (3 connections now open) m31002| Wed Jan 16 17:46:48.111 [initandlisten] connection accepted from 127.0.0.3:48439 #91 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1453, "optime" : { "t" : 1358376409000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:46:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1442, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1442, "optime" : { "t" : 1358376408000, "i" : 8113 }, "optimeDate" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1452, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1440, "optime" : { "t" : 1358376408000, "i" : 1401 }, "optimeDate" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1439, "optime" : { "t" : 1358376408000, "i" : 6885 }, "optimeDate" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1439, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1451, "optime" : { "t" : 1358376409000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:46:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:49.509 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:50.189 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:51.510 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:46:52.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1457, "optime" : { "t" : 1358376413000, "i" : 2638 }, "optimeDate" : ISODate("2013-01-16T22:46:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1446, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1446, "optime" : { "t" : 1358376409000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:46:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1456, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1444, "optime" : { "t" : 1358376409000, "i" : 521 }, "optimeDate" : ISODate("2013-01-16T22:46:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1443, "optime" : { "t" : 1358376412000, "i" : 5092 }, "optimeDate" : ISODate("2013-01-16T22:46:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1443, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1455, "optime" : { "t" : 1358376413000, "i" : 2638 }, "optimeDate" : ISODate("2013-01-16T22:46:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:53.510 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:46:55.189 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:46:55.511 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1461, "optime" : { "t" : 1358376417000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:46:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1450, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1450, "optime" : { "t" : 1358376416000, "i" : 148 }, "optimeDate" : ISODate("2013-01-16T22:46:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1460, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1448, "optime" : { "t" : 1358376413000, "i" : 2638 }, "optimeDate" : ISODate("2013-01-16T22:46:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:46:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1447, "optime" : { "t" : 1358376416000, "i" : 2959 }, "optimeDate" : ISODate("2013-01-16T22:46:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1447, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:46:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1459, "optime" : { "t" : 1358376417000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:46:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:46:57.511 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:46:59.511 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:00.190 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:47:01.512 [conn88] end connection 127.0.0.2:37346 (2 connections now open) m31001| Wed Jan 16 17:47:01.512 [initandlisten] connection accepted from 127.0.0.2:58712 #90 (3 connections now open) m31000| Wed Jan 16 17:47:01.512 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1465, "optime" : { "t" : 1358376421000, "i" : 5267 }, "optimeDate" : ISODate("2013-01-16T22:47:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1454, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1454, "optime" : { "t" : 1358376421000, "i" : 3710 }, "optimeDate" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1464, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1452, "optime" : { "t" : 1358376417000, "i" : 4067 }, "optimeDate" : ISODate("2013-01-16T22:46:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1451, "optime" : { "t" : 1358376420000, "i" : 2068 }, "optimeDate" : ISODate("2013-01-16T22:47:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1451, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:46:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1463, "optime" : { "t" : 1358376421000, "i" : 5267 }, "optimeDate" : ISODate("2013-01-16T22:47:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:03.512 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:04.747 [conn59] end connection 127.0.0.4:45611 (4 connections now open) m31000| Wed Jan 16 17:47:04.747 [initandlisten] connection accepted from 127.0.0.4:59274 #60 (6 connections now open) m31002| Wed Jan 16 17:47:05.190 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:47:05.513 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1469, "optime" : { "t" : 1358376425000, "i" : 6511 }, "optimeDate" : ISODate("2013-01-16T22:47:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1458, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1458, "optime" : { "t" : 1358376421000, "i" : 5267 }, "optimeDate" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1468, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1456, "optime" : { "t" : 1358376421000, "i" : 5267 }, "optimeDate" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:05Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1455, "optime" : { "t" : 1358376424000, "i" : 1094 }, "optimeDate" : ISODate("2013-01-16T22:47:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1455, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1467, "optime" : { "t" : 1358376425000, "i" : 6094 }, "optimeDate" : ISODate("2013-01-16T22:47:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:07.513 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:09.513 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1473, "optime" : { "t" : 1358376429000, "i" : 7943 }, "optimeDate" : ISODate("2013-01-16T22:47:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1462, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1462, "optime" : { "t" : 1358376428000, "i" : 1409 }, "optimeDate" : ISODate("2013-01-16T22:47:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1472, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1460, "optime" : { "t" : 1358376425000, "i" : 6511 }, "optimeDate" : ISODate("2013-01-16T22:47:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1459, "optime" : { "t" : 1358376428000, "i" : 217 }, "optimeDate" : ISODate("2013-01-16T22:47:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1459, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:08Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1471, "optime" : { "t" : 1358376429000, "i" : 7530 }, "optimeDate" : ISODate("2013-01-16T22:47:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:47:10.191 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:47:11.514 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:13.514 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1478, "optime" : { "t" : 1358376434000, "i" : 166 }, "optimeDate" : ISODate("2013-01-16T22:47:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1467, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1467, "optime" : { "t" : 1358376433000, "i" : 8743 }, "optimeDate" : ISODate("2013-01-16T22:47:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1477, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1465, "optime" : { "t" : 1358376429000, "i" : 7943 }, "optimeDate" : ISODate("2013-01-16T22:47:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:12Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1464, "optime" : { "t" : 1358376429000, "i" : 7943 }, "optimeDate" : ISODate("2013-01-16T22:47:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1464, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1476, "optime" : { "t" : 1358376434000, "i" : 166 }, "optimeDate" : ISODate("2013-01-16T22:47:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:47:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:47:15.192 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:47:15.514 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:17.514 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:17.972 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:393028 196ms STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 17:47:18.116 [conn91] end connection 127.0.0.3:48439 (3 connections now open) m31002| Wed Jan 16 17:47:18.116 [initandlisten] connection accepted from 127.0.0.3:57396 #92 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1482, "optime" : { "t" : 1358376438000, "i" : 2719 }, "optimeDate" : ISODate("2013-01-16T22:47:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1471, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1471, "optime" : { "t" : 1358376434000, "i" : 166 }, "optimeDate" : ISODate("2013-01-16T22:47:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1481, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1469, "optime" : { "t" : 1358376438000, "i" : 1330 }, "optimeDate" : ISODate("2013-01-16T22:47:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1468, "optime" : { "t" : 1358376434000, "i" : 166 }, "optimeDate" : ISODate("2013-01-16T22:47:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1468, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1480, "optime" : { "t" : 1358376438000, "i" : 2612 }, "optimeDate" : ISODate("2013-01-16T22:47:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:19.515 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:20.192 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:47:21.441 [conn90] end connection 127.0.0.2:35319 (3 connections now open) m31000| Wed Jan 16 17:47:21.515 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:21.941 [initandlisten] connection accepted from 127.0.0.2:51931 #93 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1486, "optime" : { "t" : 1358376442000, "i" : 2164 }, "optimeDate" : ISODate("2013-01-16T22:47:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1475, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1475, "optime" : { "t" : 1358376441000, "i" : 7378 }, "optimeDate" : ISODate("2013-01-16T22:47:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1485, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1473, "optime" : { "t" : 1358376442000, "i" : 969 }, "optimeDate" : ISODate("2013-01-16T22:47:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1472, "optime" : { "t" : 1358376438000, "i" : 2719 }, "optimeDate" : ISODate("2013-01-16T22:47:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1472, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1484, "optime" : { "t" : 1358376442000, "i" : 2164 }, "optimeDate" : ISODate("2013-01-16T22:47:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:23.515 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:47:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:47:25.193 [conn89] end connection 127.0.0.4:45104 (2 connections now open) m31000| Wed Jan 16 17:47:25.516 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:47:25.693 [initandlisten] connection accepted from 127.0.0.4:59461 #91 (3 connections now open) m31002| Wed Jan 16 17:47:25.693 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1490, "optime" : { "t" : 1358376446000, "i" : 3478 }, "optimeDate" : ISODate("2013-01-16T22:47:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1479, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1479, "optime" : { "t" : 1358376442000, "i" : 2164 }, "optimeDate" : ISODate("2013-01-16T22:47:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1489, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1477, "optime" : { "t" : 1358376446000, "i" : 587 }, "optimeDate" : ISODate("2013-01-16T22:47:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1476, "optime" : { "t" : 1358376442000, "i" : 2164 }, "optimeDate" : ISODate("2013-01-16T22:47:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1476, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1488, "optime" : { "t" : 1358376446000, "i" : 3478 }, "optimeDate" : ISODate("2013-01-16T22:47:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:27.516 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:29.516 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1494, "optime" : { "t" : 1358376450000, "i" : 5758 }, "optimeDate" : ISODate("2013-01-16T22:47:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1483, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1483, "optime" : { "t" : 1358376449000, "i" : 16 }, "optimeDate" : ISODate("2013-01-16T22:47:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1493, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1481, "optime" : { "t" : 1358376450000, "i" : 1588 }, "optimeDate" : ISODate("2013-01-16T22:47:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1480, "optime" : { "t" : 1358376446000, "i" : 3478 }, "optimeDate" : ISODate("2013-01-16T22:47:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1480, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1492, "optime" : { "t" : 1358376450000, "i" : 5758 }, "optimeDate" : ISODate("2013-01-16T22:47:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:47:30.694 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:47:31.517 [conn90] end connection 127.0.0.2:58712 (2 connections now open) m31001| Wed Jan 16 17:47:31.517 [initandlisten] connection accepted from 127.0.0.2:54085 #92 (3 connections now open) m31000| Wed Jan 16 17:47:31.517 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:33.518 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1498, "optime" : { "t" : 1358376454000, "i" : 6822 }, "optimeDate" : ISODate("2013-01-16T22:47:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1487, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1487, "optime" : { "t" : 1358376454000, "i" : 4054 }, "optimeDate" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1497, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1485, "optime" : { "t" : 1358376454000, "i" : 1340 }, "optimeDate" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1484, "optime" : { "t" : 1358376450000, "i" : 5758 }, "optimeDate" : ISODate("2013-01-16T22:47:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1484, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1496, "optime" : { "t" : 1358376454000, "i" : 6822 }, "optimeDate" : ISODate("2013-01-16T22:47:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:34.753 [conn60] end connection 127.0.0.4:59274 (4 connections now open) m31000| Wed Jan 16 17:47:34.753 [initandlisten] connection accepted from 127.0.0.4:44780 #61 (5 connections now open) m31000| Wed Jan 16 17:47:35.518 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:35.695 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:47:37.518 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1502, "optime" : { "t" : 1358376458000, "i" : 7957 }, "optimeDate" : ISODate("2013-01-16T22:47:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1491, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1491, "optime" : { "t" : 1358376454000, "i" : 6822 }, "optimeDate" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1501, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1489, "optime" : { "t" : 1358376458000, "i" : 1285 }, "optimeDate" : ISODate("2013-01-16T22:47:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1488, "optime" : { "t" : 1358376458000, "i" : 6590 }, "optimeDate" : ISODate("2013-01-16T22:47:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1488, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1500, "optime" : { "t" : 1358376458000, "i" : 7957 }, "optimeDate" : ISODate("2013-01-16T22:47:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:39.519 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:40.695 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:47:41.519 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1506, "optime" : { "t" : 1358376462000, "i" : 9396 }, "optimeDate" : ISODate("2013-01-16T22:47:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1495, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1495, "optime" : { "t" : 1358376461000, "i" : 135 }, "optimeDate" : ISODate("2013-01-16T22:47:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1505, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1493, "optime" : { "t" : 1358376462000, "i" : 1137 }, "optimeDate" : ISODate("2013-01-16T22:47:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1492, "optime" : { "t" : 1358376462000, "i" : 6678 }, "optimeDate" : ISODate("2013-01-16T22:47:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1492, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1504, "optime" : { "t" : 1358376462000, "i" : 9396 }, "optimeDate" : ISODate("2013-01-16T22:47:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:43.519 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:45.520 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:45.696 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:47:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1511, "optime" : { "t" : 1358376467000, "i" : 1536 }, "optimeDate" : ISODate("2013-01-16T22:47:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1500, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1500, "optime" : { "t" : 1358376466000, "i" : 7193 }, "optimeDate" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1510, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1498, "optime" : { "t" : 1358376466000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1497, "optime" : { "t" : 1358376466000, "i" : 6014 }, "optimeDate" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1497, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1509, "optime" : { "t" : 1358376467000, "i" : 1306 }, "optimeDate" : ISODate("2013-01-16T22:47:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:47.520 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:48.121 [conn92] end connection 127.0.0.3:57396 (3 connections now open) m31002| Wed Jan 16 17:47:48.121 [initandlisten] connection accepted from 127.0.0.3:47064 #94 (4 connections now open) m31000| Wed Jan 16 17:47:49.521 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:50.697 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1515, "optime" : { "t" : 1358376471000, "i" : 2755 }, "optimeDate" : ISODate("2013-01-16T22:47:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1504, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1504, "optime" : { "t" : 1358376467000, "i" : 1536 }, "optimeDate" : ISODate("2013-01-16T22:47:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1514, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1502, "optime" : { "t" : 1358376467000, "i" : 1536 }, "optimeDate" : ISODate("2013-01-16T22:47:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1501, "optime" : { "t" : 1358376470000, "i" : 4570 }, "optimeDate" : ISODate("2013-01-16T22:47:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1501, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1513, "optime" : { "t" : 1358376471000, "i" : 2755 }, "optimeDate" : ISODate("2013-01-16T22:47:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:51.521 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:47:53.521 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1519, "optime" : { "t" : 1358376475000, "i" : 4038 }, "optimeDate" : ISODate("2013-01-16T22:47:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1508, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1508, "optime" : { "t" : 1358376474000, "i" : 618 }, "optimeDate" : ISODate("2013-01-16T22:47:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1518, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1506, "optime" : { "t" : 1358376471000, "i" : 2755 }, "optimeDate" : ISODate("2013-01-16T22:47:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1505, "optime" : { "t" : 1358376474000, "i" : 3630 }, "optimeDate" : ISODate("2013-01-16T22:47:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1505, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1517, "optime" : { "t" : 1358376475000, "i" : 4038 }, "optimeDate" : ISODate("2013-01-16T22:47:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:47:55.522 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:55.698 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:47:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:47:57.522 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:47:59.446 [conn93] end connection 127.0.0.2:51931 (3 connections now open) m31000| Wed Jan 16 17:47:59.522 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1523, "optime" : { "t" : 1358376479000, "i" : 5156 }, "optimeDate" : ISODate("2013-01-16T22:47:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1512, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1512, "optime" : { "t" : 1358376475000, "i" : 4038 }, "optimeDate" : ISODate("2013-01-16T22:47:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1522, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1510, "optime" : { "t" : 1358376475000, "i" : 4038 }, "optimeDate" : ISODate("2013-01-16T22:47:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:47:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:47:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1509, "optime" : { "t" : 1358376478000, "i" : 2745 }, "optimeDate" : ISODate("2013-01-16T22:47:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1509, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1521, "optime" : { "t" : 1358376479000, "i" : 5156 }, "optimeDate" : ISODate("2013-01-16T22:47:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:47:59.946 [initandlisten] connection accepted from 127.0.0.2:34550 #95 (4 connections now open) m31002| Wed Jan 16 17:48:00.698 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:48:01.523 [conn92] end connection 127.0.0.2:54085 (2 connections now open) m31001| Wed Jan 16 17:48:01.523 [initandlisten] connection accepted from 127.0.0.2:54686 #93 (3 connections now open) m31000| Wed Jan 16 17:48:01.523 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:48:03.198 [conn91] end connection 127.0.0.4:59461 (2 connections now open) m31000| Wed Jan 16 17:48:03.523 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1527, "optime" : { "t" : 1358376483000, "i" : 5675 }, "optimeDate" : ISODate("2013-01-16T22:48:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1516, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1516, "optime" : { "t" : 1358376479000, "i" : 5156 }, "optimeDate" : ISODate("2013-01-16T22:47:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:47:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1526, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1514, "optime" : { "t" : 1358376479000, "i" : 5156 }, "optimeDate" : ISODate("2013-01-16T22:47:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1513, "optime" : { "t" : 1358376482000, "i" : 1696 }, "optimeDate" : ISODate("2013-01-16T22:48:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1513, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1525, "optime" : { "t" : 1358376483000, "i" : 5529 }, "optimeDate" : ISODate("2013-01-16T22:48:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:48:03.699 [initandlisten] connection accepted from 127.0.0.4:48863 #94 (3 connections now open) m31000| Wed Jan 16 17:48:04.758 [conn61] end connection 127.0.0.4:44780 (4 connections now open) m31000| Wed Jan 16 17:48:04.758 [initandlisten] connection accepted from 127.0.0.4:45889 #62 (5 connections now open) m31000| Wed Jan 16 17:48:05.524 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:06.199 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:07.524 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1531, "optime" : { "t" : 1358376487000, "i" : 7009 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1520, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1520, "optime" : { "t" : 1358376487000, "i" : 3434 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1530, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1518, "optime" : { "t" : 1358376483000, "i" : 5675 }, "optimeDate" : ISODate("2013-01-16T22:48:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1517, "optime" : { "t" : 1358376486000, "i" : 713 }, "optimeDate" : ISODate("2013-01-16T22:48:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1517, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1529, "optime" : { "t" : 1358376487000, "i" : 7009 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:09.524 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:11.200 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:11.525 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1535, "optime" : { "t" : 1358376491000, "i" : 8685 }, "optimeDate" : ISODate("2013-01-16T22:48:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1524, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1524, "optime" : { "t" : 1358376487000, "i" : 7009 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1534, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1522, "optime" : { "t" : 1358376487000, "i" : 7009 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1521, "optime" : { "t" : 1358376487000, "i" : 7009 }, "optimeDate" : ISODate("2013-01-16T22:48:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1521, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1533, "optime" : { "t" : 1358376491000, "i" : 8685 }, "optimeDate" : ISODate("2013-01-16T22:48:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:13.525 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:48:15.525 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1540, "optime" : { "t" : 1358376496000, "i" : 1023 }, "optimeDate" : ISODate("2013-01-16T22:48:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1529, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1529, "optime" : { "t" : 1358376491000, "i" : 8685 }, "optimeDate" : ISODate("2013-01-16T22:48:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1539, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1527, "optime" : { "t" : 1358376491000, "i" : 8685 }, "optimeDate" : ISODate("2013-01-16T22:48:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1526, "optime" : { "t" : 1358376491000, "i" : 8685 }, "optimeDate" : ISODate("2013-01-16T22:48:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1526, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1538, "optime" : { "t" : 1358376496000, "i" : 732 }, "optimeDate" : ISODate("2013-01-16T22:48:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:48:16.201 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:17.526 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:18.127 [conn94] end connection 127.0.0.3:47064 (3 connections now open) m31002| Wed Jan 16 17:48:18.127 [initandlisten] connection accepted from 127.0.0.3:36023 #96 (4 connections now open) m31001| Wed Jan 16 17:48:18.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:48:19.526 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1544, "optime" : { "t" : 1358376500000, "i" : 1864 }, "optimeDate" : ISODate("2013-01-16T22:48:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1533, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1533, "optime" : { "t" : 1358376499000, "i" : 6755 }, "optimeDate" : ISODate("2013-01-16T22:48:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1543, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1531, "optime" : { "t" : 1358376500000, "i" : 1369 }, "optimeDate" : ISODate("2013-01-16T22:48:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1530, "optime" : { "t" : 1358376496000, "i" : 1023 }, "optimeDate" : ISODate("2013-01-16T22:48:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1530, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1542, "optime" : { "t" : 1358376500000, "i" : 1513 }, "optimeDate" : ISODate("2013-01-16T22:48:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:48:21.201 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:21.527 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:48:23.527 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1548, "optime" : { "t" : 1358376504000, "i" : 2559 }, "optimeDate" : ISODate("2013-01-16T22:48:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1537, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1537, "optime" : { "t" : 1358376500000, "i" : 1864 }, "optimeDate" : ISODate("2013-01-16T22:48:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1547, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1535, "optime" : { "t" : 1358376504000, "i" : 187 }, "optimeDate" : ISODate("2013-01-16T22:48:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1534, "optime" : { "t" : 1358376500000, "i" : 1864 }, "optimeDate" : ISODate("2013-01-16T22:48:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1534, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1546, "optime" : { "t" : 1358376504000, "i" : 2184 }, "optimeDate" : ISODate("2013-01-16T22:48:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:25.527 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:26.202 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:27.527 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1552, "optime" : { "t" : 1358376508000, "i" : 4695 }, "optimeDate" : ISODate("2013-01-16T22:48:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1541, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1541, "optime" : { "t" : 1358376507000, "i" : 1564 }, "optimeDate" : ISODate("2013-01-16T22:48:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1551, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1539, "optime" : { "t" : 1358376508000, "i" : 1296 }, "optimeDate" : ISODate("2013-01-16T22:48:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1538, "optime" : { "t" : 1358376504000, "i" : 2559 }, "optimeDate" : ISODate("2013-01-16T22:48:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1538, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1550, "optime" : { "t" : 1358376508000, "i" : 4695 }, "optimeDate" : ISODate("2013-01-16T22:48:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:48:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:48:29.528 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:31.203 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:48:31.528 [conn93] end connection 127.0.0.2:54686 (2 connections now open) m31001| Wed Jan 16 17:48:31.528 [initandlisten] connection accepted from 127.0.0.2:39915 #95 (3 connections now open) m31000| Wed Jan 16 17:48:31.528 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1556, "optime" : { "t" : 1358376512000, "i" : 4206 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1545, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1545, "optime" : { "t" : 1358376512000, "i" : 3689 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1555, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1543, "optime" : { "t" : 1358376512000, "i" : 296 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1542, "optime" : { "t" : 1358376508000, "i" : 4695 }, "optimeDate" : ISODate("2013-01-16T22:48:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1542, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1554, "optime" : { "t" : 1358376512000, "i" : 4206 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:33.529 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:48:34.764 [conn62] end connection 127.0.0.4:45889 (4 connections now open) m31000| Wed Jan 16 17:48:34.764 [initandlisten] connection accepted from 127.0.0.4:43489 #63 (5 connections now open) m31000| Wed Jan 16 17:48:35.529 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:36.203 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1560, "optime" : { "t" : 1358376516000, "i" : 5478 }, "optimeDate" : ISODate("2013-01-16T22:48:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1549, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1549, "optime" : { "t" : 1358376512000, "i" : 4206 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1559, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1547, "optime" : { "t" : 1358376516000, "i" : 1107 }, "optimeDate" : ISODate("2013-01-16T22:48:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1546, "optime" : { "t" : 1358376512000, "i" : 4206 }, "optimeDate" : ISODate("2013-01-16T22:48:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1546, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1558, "optime" : { "t" : 1358376516000, "i" : 5478 }, "optimeDate" : ISODate("2013-01-16T22:48:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:48:37.451 [conn95] end connection 127.0.0.2:34550 (3 connections now open) m31000| Wed Jan 16 17:48:37.529 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:37.952 [initandlisten] connection accepted from 127.0.0.2:60026 #97 (4 connections now open) m31000| Wed Jan 16 17:48:39.530 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1564, "optime" : { "t" : 1358376520000, "i" : 6583 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1553, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1553, "optime" : { "t" : 1358376520000, "i" : 3707 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1551, "optime" : { "t" : 1358376520000, "i" : 1199 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1550, "optime" : { "t" : 1358376520000, "i" : 6567 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1550, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1562, "optime" : { "t" : 1358376520000, "i" : 6583 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:48:41.204 [conn94] end connection 127.0.0.4:48863 (2 connections now open) m31000| Wed Jan 16 17:48:41.530 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:48:41.704 [initandlisten] connection accepted from 127.0.0.4:45282 #96 (3 connections now open) m31002| Wed Jan 16 17:48:41.704 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:43.531 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1568, "optime" : { "t" : 1358376524000, "i" : 8425 }, "optimeDate" : ISODate("2013-01-16T22:48:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1557, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1557, "optime" : { "t" : 1358376520000, "i" : 6583 }, "optimeDate" : ISODate("2013-01-16T22:48:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1568, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1556, "optime" : { "t" : 1358376524000, "i" : 1247 }, "optimeDate" : ISODate("2013-01-16T22:48:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1555, "optime" : { "t" : 1358376524000, "i" : 6803 }, "optimeDate" : ISODate("2013-01-16T22:48:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1555, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1567, "optime" : { "t" : 1358376524000, "i" : 8228 }, "optimeDate" : ISODate("2013-01-16T22:48:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:45.531 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:46.705 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:47.532 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:48.131 [conn96] end connection 127.0.0.3:36023 (3 connections now open) m31002| Wed Jan 16 17:48:48.132 [initandlisten] connection accepted from 127.0.0.3:33121 #98 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1573, "optime" : { "t" : 1358376529000, "i" : 1904 }, "optimeDate" : ISODate("2013-01-16T22:48:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1562, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1562, "optime" : { "t" : 1358376524000, "i" : 8425 }, "optimeDate" : ISODate("2013-01-16T22:48:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1572, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1560, "optime" : { "t" : 1358376528000, "i" : 320 }, "optimeDate" : ISODate("2013-01-16T22:48:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1559, "optime" : { "t" : 1358376528000, "i" : 6379 }, "optimeDate" : ISODate("2013-01-16T22:48:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1559, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1571, "optime" : { "t" : 1358376529000, "i" : 1887 }, "optimeDate" : ISODate("2013-01-16T22:48:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:49.532 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:48:50.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:48:51.532 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:51.705 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:48:52.975 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:299584 149ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1577, "optime" : { "t" : 1358376533000, "i" : 3449 }, "optimeDate" : ISODate("2013-01-16T22:48:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1566, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1566, "optime" : { "t" : 1358376532000, "i" : 5878 }, "optimeDate" : ISODate("2013-01-16T22:48:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1576, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1564, "optime" : { "t" : 1358376529000, "i" : 1904 }, "optimeDate" : ISODate("2013-01-16T22:48:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1563, "optime" : { "t" : 1358376532000, "i" : 5441 }, "optimeDate" : ISODate("2013-01-16T22:48:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1575, "optime" : { "t" : 1358376533000, "i" : 3241 }, "optimeDate" : ISODate("2013-01-16T22:48:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:53.533 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:48:55.533 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:48:56.706 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1581, "optime" : { "t" : 1358376537000, "i" : 3552 }, "optimeDate" : ISODate("2013-01-16T22:48:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1570, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1570, "optime" : { "t" : 1358376533000, "i" : 3449 }, "optimeDate" : ISODate("2013-01-16T22:48:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1580, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1568, "optime" : { "t" : 1358376533000, "i" : 3449 }, "optimeDate" : ISODate("2013-01-16T22:48:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:48:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:48:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1567, "optime" : { "t" : 1358376536000, "i" : 4659 }, "optimeDate" : ISODate("2013-01-16T22:48:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1567, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1579, "optime" : { "t" : 1358376537000, "i" : 3308 }, "optimeDate" : ISODate("2013-01-16T22:48:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:48:57.534 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:48:59.534 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:49:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 m31001| Wed Jan 16 17:49:01.534 [conn95] end connection 127.0.0.2:39915 (2 connections now open) m31001| Wed Jan 16 17:49:01.535 [initandlisten] connection accepted from 127.0.0.2:43335 #97 (3 connections now open) m31000| Wed Jan 16 17:49:01.535 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1585, "optime" : { "t" : 1358376541000, "i" : 4349 }, "optimeDate" : ISODate("2013-01-16T22:49:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1574, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1574, "optime" : { "t" : 1358376540000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:49:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1584, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1572, "optime" : { "t" : 1358376537000, "i" : 3552 }, "optimeDate" : ISODate("2013-01-16T22:48:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1571, "optime" : { "t" : 1358376540000, "i" : 3453 }, "optimeDate" : ISODate("2013-01-16T22:49:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1571, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1583, "optime" : { "t" : 1358376541000, "i" : 4294 }, "optimeDate" : ISODate("2013-01-16T22:49:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:49:01.707 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:03.535 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:04.769 [conn63] end connection 127.0.0.4:43489 (4 connections now open) m31000| Wed Jan 16 17:49:04.769 [initandlisten] connection accepted from 127.0.0.4:48872 #64 (6 connections now open) m31000| Wed Jan 16 17:49:05.536 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1589, "optime" : { "t" : 1358376545000, "i" : 6317 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1578, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1578, "optime" : { "t" : 1358376545000, "i" : 3570 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1588, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1576, "optime" : { "t" : 1358376541000, "i" : 4349 }, "optimeDate" : ISODate("2013-01-16T22:49:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1575, "optime" : { "t" : 1358376544000, "i" : 1634 }, "optimeDate" : ISODate("2013-01-16T22:49:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1575, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1587, "optime" : { "t" : 1358376545000, "i" : 6317 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:49:06.708 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:07.536 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:09.536 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1593, "optime" : { "t" : 1358376549000, "i" : 8628 }, "optimeDate" : ISODate("2013-01-16T22:49:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1582, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1582, "optime" : { "t" : 1358376545000, "i" : 6317 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:48:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1592, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1580, "optime" : { "t" : 1358376545000, "i" : 6317 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1579, "optime" : { "t" : 1358376545000, "i" : 6317 }, "optimeDate" : ISODate("2013-01-16T22:49:05Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1579, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1591, "optime" : { "t" : 1358376549000, "i" : 7830 }, "optimeDate" : ISODate("2013-01-16T22:49:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:11.537 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:11.708 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:13.537 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1598, "optime" : { "t" : 1358376554000, "i" : 150 }, "optimeDate" : ISODate("2013-01-16T22:49:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1587, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1587, "optime" : { "t" : 1358376552000, "i" : 70 }, "optimeDate" : ISODate("2013-01-16T22:49:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1597, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1585, "optime" : { "t" : 1358376549000, "i" : 8628 }, "optimeDate" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1584, "optime" : { "t" : 1358376549000, "i" : 8628 }, "optimeDate" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:12Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1584, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1596, "optime" : { "t" : 1358376554000, "i" : 118 }, "optimeDate" : ISODate("2013-01-16T22:49:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:49:15.456 [conn97] end connection 127.0.0.2:60026 (3 connections now open) m31000| Wed Jan 16 17:49:15.537 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:15.957 [initandlisten] connection accepted from 127.0.0.2:40404 #99 (4 connections now open) m31002| Wed Jan 16 17:49:16.709 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:17.538 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:18.137 [conn98] end connection 127.0.0.3:33121 (3 connections now open) m31002| Wed Jan 16 17:49:18.137 [initandlisten] connection accepted from 127.0.0.3:46839 #100 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1602, "optime" : { "t" : 1358376558000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:49:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1591, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1591, "optime" : { "t" : 1358376554000, "i" : 150 }, "optimeDate" : ISODate("2013-01-16T22:49:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1601, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1589, "optime" : { "t" : 1358376558000, "i" : 992 }, "optimeDate" : ISODate("2013-01-16T22:49:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1588, "optime" : { "t" : 1358376554000, "i" : 150 }, "optimeDate" : ISODate("2013-01-16T22:49:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1588, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1600, "optime" : { "t" : 1358376558000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:49:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:49:19.209 [conn96] end connection 127.0.0.4:45282 (2 connections now open) m31000| Wed Jan 16 17:49:19.538 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:49:19.709 [initandlisten] connection accepted from 127.0.0.4:54712 #98 (3 connections now open) m31000| Wed Jan 16 17:49:21.538 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:22.210 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1606, "optime" : { "t" : 1358376562000, "i" : 3792 }, "optimeDate" : ISODate("2013-01-16T22:49:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1595, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1595, "optime" : { "t" : 1358376558000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:49:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1605, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1593, "optime" : { "t" : 1358376562000, "i" : 949 }, "optimeDate" : ISODate("2013-01-16T22:49:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1592, "optime" : { "t" : 1358376558000, "i" : 2110 }, "optimeDate" : ISODate("2013-01-16T22:49:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1592, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1604, "optime" : { "t" : 1358376562000, "i" : 3792 }, "optimeDate" : ISODate("2013-01-16T22:49:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:49:22.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:49:23.538 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:25.539 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1610, "optime" : { "t" : 1358376566000, "i" : 4790 }, "optimeDate" : ISODate("2013-01-16T22:49:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1599, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1599, "optime" : { "t" : 1358376565000, "i" : 5082 }, "optimeDate" : ISODate("2013-01-16T22:49:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1609, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1597, "optime" : { "t" : 1358376566000, "i" : 1345 }, "optimeDate" : ISODate("2013-01-16T22:49:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1596, "optime" : { "t" : 1358376562000, "i" : 3792 }, "optimeDate" : ISODate("2013-01-16T22:49:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1596, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1608, "optime" : { "t" : 1358376566000, "i" : 4790 }, "optimeDate" : ISODate("2013-01-16T22:49:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:49:27.210 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:27.539 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:29.540 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1614, "optime" : { "t" : 1358376570000, "i" : 5849 }, "optimeDate" : ISODate("2013-01-16T22:49:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1603, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1603, "optime" : { "t" : 1358376566000, "i" : 4790 }, "optimeDate" : ISODate("2013-01-16T22:49:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1613, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1601, "optime" : { "t" : 1358376570000, "i" : 1016 }, "optimeDate" : ISODate("2013-01-16T22:49:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1600, "optime" : { "t" : 1358376566000, "i" : 4790 }, "optimeDate" : ISODate("2013-01-16T22:49:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1600, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1612, "optime" : { "t" : 1358376570000, "i" : 5849 }, "optimeDate" : ISODate("2013-01-16T22:49:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:49:31.540 [conn97] end connection 127.0.0.2:43335 (2 connections now open) m31001| Wed Jan 16 17:49:31.540 [initandlisten] connection accepted from 127.0.0.2:34339 #99 (3 connections now open) m31000| Wed Jan 16 17:49:31.540 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:32.211 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:49:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:49:33.541 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:34.774 [conn64] end connection 127.0.0.4:48872 (4 connections now open) m31000| Wed Jan 16 17:49:34.774 [initandlisten] connection accepted from 127.0.0.4:46141 #65 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1618, "optime" : { "t" : 1358376574000, "i" : 6916 }, "optimeDate" : ISODate("2013-01-16T22:49:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1607, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1607, "optime" : { "t" : 1358376570000, "i" : 5849 }, "optimeDate" : ISODate("2013-01-16T22:49:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1617, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1605, "optime" : { "t" : 1358376574000, "i" : 1073 }, "optimeDate" : ISODate("2013-01-16T22:49:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1604, "optime" : { "t" : 1358376574000, "i" : 6616 }, "optimeDate" : ISODate("2013-01-16T22:49:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1604, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1616, "optime" : { "t" : 1358376574000, "i" : 6866 }, "optimeDate" : ISODate("2013-01-16T22:49:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:35.541 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:37.212 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:37.541 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:37.997 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:209685 104ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1622, "optime" : { "t" : 1358376578000, "i" : 9376 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1611, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1611, "optime" : { "t" : 1358376578000, "i" : 5632 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1621, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1609, "optime" : { "t" : 1358376578000, "i" : 1723 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1608, "optime" : { "t" : 1358376578000, "i" : 8364 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1608, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1620, "optime" : { "t" : 1358376578000, "i" : 9376 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:39.542 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:41.542 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:42.212 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1627, "optime" : { "t" : 1358376583000, "i" : 932 }, "optimeDate" : ISODate("2013-01-16T22:49:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1616, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1616, "optime" : { "t" : 1358376578000, "i" : 9376 }, "optimeDate" : ISODate("2013-01-16T22:49:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1626, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1614, "optime" : { "t" : 1358376582000, "i" : 1408 }, "optimeDate" : ISODate("2013-01-16T22:49:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1613, "optime" : { "t" : 1358376582000, "i" : 6778 }, "optimeDate" : ISODate("2013-01-16T22:49:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1613, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1625, "optime" : { "t" : 1358376583000, "i" : 932 }, "optimeDate" : ISODate("2013-01-16T22:49:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:43.543 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:49:45.543 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:47.213 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1631, "optime" : { "t" : 1358376587000, "i" : 2460 }, "optimeDate" : ISODate("2013-01-16T22:49:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1620, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1620, "optime" : { "t" : 1358376583000, "i" : 932 }, "optimeDate" : ISODate("2013-01-16T22:49:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1630, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1618, "optime" : { "t" : 1358376586000, "i" : 346 }, "optimeDate" : ISODate("2013-01-16T22:49:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1617, "optime" : { "t" : 1358376586000, "i" : 5418 }, "optimeDate" : ISODate("2013-01-16T22:49:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1617, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1629, "optime" : { "t" : 1358376587000, "i" : 2289 }, "optimeDate" : ISODate("2013-01-16T22:49:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:47.544 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:48.142 [conn100] end connection 127.0.0.3:46839 (3 connections now open) m31002| Wed Jan 16 17:49:48.142 [initandlisten] connection accepted from 127.0.0.3:37494 #101 (4 connections now open) m31000| Wed Jan 16 17:49:49.544 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1635, "optime" : { "t" : 1358376591000, "i" : 3874 }, "optimeDate" : ISODate("2013-01-16T22:49:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1624, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1624, "optime" : { "t" : 1358376590000, "i" : 5975 }, "optimeDate" : ISODate("2013-01-16T22:49:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1634, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1622, "optime" : { "t" : 1358376587000, "i" : 2460 }, "optimeDate" : ISODate("2013-01-16T22:49:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1621, "optime" : { "t" : 1358376590000, "i" : 4374 }, "optimeDate" : ISODate("2013-01-16T22:49:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1621, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1633, "optime" : { "t" : 1358376591000, "i" : 3730 }, "optimeDate" : ISODate("2013-01-16T22:49:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:49:51.544 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:52.214 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:49:53.461 [conn99] end connection 127.0.0.2:40404 (3 connections now open) m31000| Wed Jan 16 17:49:53.545 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:49:53.961 [initandlisten] connection accepted from 127.0.0.2:36533 #102 (4 connections now open) m31001| Wed Jan 16 17:49:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:49:55.545 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1639, "optime" : { "t" : 1358376595000, "i" : 4536 }, "optimeDate" : ISODate("2013-01-16T22:49:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1628, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1628, "optime" : { "t" : 1358376591000, "i" : 3874 }, "optimeDate" : ISODate("2013-01-16T22:49:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1638, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1626, "optime" : { "t" : 1358376591000, "i" : 3874 }, "optimeDate" : ISODate("2013-01-16T22:49:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1625, "optime" : { "t" : 1358376594000, "i" : 3170 }, "optimeDate" : ISODate("2013-01-16T22:49:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1625, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1637, "optime" : { "t" : 1358376595000, "i" : 4536 }, "optimeDate" : ISODate("2013-01-16T22:49:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:49:57.214 [conn98] end connection 127.0.0.4:54712 (2 connections now open) m31000| Wed Jan 16 17:49:57.545 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:49:57.714 [initandlisten] connection accepted from 127.0.0.4:46020 #100 (3 connections now open) m31002| Wed Jan 16 17:49:57.715 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:49:59.546 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1643, "optime" : { "t" : 1358376599000, "i" : 6150 }, "optimeDate" : ISODate("2013-01-16T22:49:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1632, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1632, "optime" : { "t" : 1358376598000, "i" : 2631 }, "optimeDate" : ISODate("2013-01-16T22:49:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1642, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1630, "optime" : { "t" : 1358376595000, "i" : 4536 }, "optimeDate" : ISODate("2013-01-16T22:49:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:49:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1629, "optime" : { "t" : 1358376598000, "i" : 1456 }, "optimeDate" : ISODate("2013-01-16T22:49:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1629, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:49:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1641, "optime" : { "t" : 1358376599000, "i" : 5473 }, "optimeDate" : ISODate("2013-01-16T22:49:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:50:01.546 [conn99] end connection 127.0.0.2:34339 (2 connections now open) m31001| Wed Jan 16 17:50:01.546 [initandlisten] connection accepted from 127.0.0.2:37454 #101 (3 connections now open) m31000| Wed Jan 16 17:50:01.546 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:02.715 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:02.982 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:276703 138ms m31000| Wed Jan 16 17:50:03.547 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1647, "optime" : { "t" : 1358376603000, "i" : 8040 }, "optimeDate" : ISODate("2013-01-16T22:50:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1636, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1636, "optime" : { "t" : 1358376599000, "i" : 6150 }, "optimeDate" : ISODate("2013-01-16T22:49:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1646, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1634, "optime" : { "t" : 1358376599000, "i" : 6150 }, "optimeDate" : ISODate("2013-01-16T22:49:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1633, "optime" : { "t" : 1358376602000, "i" : 1000 }, "optimeDate" : ISODate("2013-01-16T22:50:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1633, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1645, "optime" : { "t" : 1358376603000, "i" : 7803 }, "optimeDate" : ISODate("2013-01-16T22:50:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:04.779 [conn65] end connection 127.0.0.4:46141 (4 connections now open) m31000| Wed Jan 16 17:50:04.779 [initandlisten] connection accepted from 127.0.0.4:36512 #66 (5 connections now open) m31001| Wed Jan 16 17:50:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:50:05.547 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:07.245 [journal] old journal file will be removed: /data/db/testReplSet-0/journal/j._1 m31000| Wed Jan 16 17:50:07.313 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:278339 139ms m31000| Wed Jan 16 17:50:07.547 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:07.716 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1651, "optime" : { "t" : 1358376607000, "i" : 8411 }, "optimeDate" : ISODate("2013-01-16T22:50:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1640, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1640, "optime" : { "t" : 1358376603000, "i" : 8040 }, "optimeDate" : ISODate("2013-01-16T22:50:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1650, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1638, "optime" : { "t" : 1358376603000, "i" : 8040 }, "optimeDate" : ISODate("2013-01-16T22:50:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:07Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1637, "optime" : { "t" : 1358376603000, "i" : 8040 }, "optimeDate" : ISODate("2013-01-16T22:50:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1637, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1649, "optime" : { "t" : 1358376607000, "i" : 8411 }, "optimeDate" : ISODate("2013-01-16T22:50:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:09.548 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:11.548 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1656, "optime" : { "t" : 1358376612000, "i" : 1879 }, "optimeDate" : ISODate("2013-01-16T22:50:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1645, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1645, "optime" : { "t" : 1358376611000, "i" : 3752 }, "optimeDate" : ISODate("2013-01-16T22:50:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1655, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1643, "optime" : { "t" : 1358376612000, "i" : 1378 }, "optimeDate" : ISODate("2013-01-16T22:50:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1642, "optime" : { "t" : 1358376607000, "i" : 8411 }, "optimeDate" : ISODate("2013-01-16T22:50:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1642, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1654, "optime" : { "t" : 1358376612000, "i" : 1879 }, "optimeDate" : ISODate("2013-01-16T22:50:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:50:12.717 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:13.548 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:50:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:50:15.549 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1660, "optime" : { "t" : 1358376616000, "i" : 2801 }, "optimeDate" : ISODate("2013-01-16T22:50:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1649, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1649, "optime" : { "t" : 1358376612000, "i" : 1879 }, "optimeDate" : ISODate("2013-01-16T22:50:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1659, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1647, "optime" : { "t" : 1358376616000, "i" : 1298 }, "optimeDate" : ISODate("2013-01-16T22:50:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1646, "optime" : { "t" : 1358376612000, "i" : 1879 }, "optimeDate" : ISODate("2013-01-16T22:50:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1646, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1658, "optime" : { "t" : 1358376616000, "i" : 2801 }, "optimeDate" : ISODate("2013-01-16T22:50:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:17.549 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:17.717 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:50:18.148 [conn101] end connection 127.0.0.3:37494 (3 connections now open) m31002| Wed Jan 16 17:50:18.148 [initandlisten] connection accepted from 127.0.0.3:52524 #103 (4 connections now open) m31000| Wed Jan 16 17:50:19.549 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1664, "optime" : { "t" : 1358376620000, "i" : 4055 }, "optimeDate" : ISODate("2013-01-16T22:50:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1653, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1653, "optime" : { "t" : 1358376616000, "i" : 2801 }, "optimeDate" : ISODate("2013-01-16T22:50:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1663, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1651, "optime" : { "t" : 1358376620000, "i" : 1242 }, "optimeDate" : ISODate("2013-01-16T22:50:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1650, "optime" : { "t" : 1358376616000, "i" : 2801 }, "optimeDate" : ISODate("2013-01-16T22:50:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1650, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1662, "optime" : { "t" : 1358376620000, "i" : 4055 }, "optimeDate" : ISODate("2013-01-16T22:50:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:21.550 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:22.718 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:23.550 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1668, "optime" : { "t" : 1358376624000, "i" : 5501 }, "optimeDate" : ISODate("2013-01-16T22:50:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1657, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1657, "optime" : { "t" : 1358376623000, "i" : 3906 }, "optimeDate" : ISODate("2013-01-16T22:50:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:49:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1667, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1655, "optime" : { "t" : 1358376624000, "i" : 1361 }, "optimeDate" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1654, "optime" : { "t" : 1358376620000, "i" : 4055 }, "optimeDate" : ISODate("2013-01-16T22:50:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1654, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1666, "optime" : { "t" : 1358376624000, "i" : 5501 }, "optimeDate" : ISODate("2013-01-16T22:50:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:25.550 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:27.551 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:27.719 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1672, "optime" : { "t" : 1358376628000, "i" : 6490 }, "optimeDate" : ISODate("2013-01-16T22:50:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1661, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1661, "optime" : { "t" : 1358376624000, "i" : 5501 }, "optimeDate" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1671, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1659, "optime" : { "t" : 1358376628000, "i" : 1356 }, "optimeDate" : ISODate("2013-01-16T22:50:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1658, "optime" : { "t" : 1358376624000, "i" : 5501 }, "optimeDate" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1658, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1670, "optime" : { "t" : 1358376628000, "i" : 6302 }, "optimeDate" : ISODate("2013-01-16T22:50:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:29.551 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:31.466 [conn102] end connection 127.0.0.2:36533 (3 connections now open) m31001| Wed Jan 16 17:50:31.551 [conn101] end connection 127.0.0.2:37454 (2 connections now open) m31001| Wed Jan 16 17:50:31.552 [initandlisten] connection accepted from 127.0.0.2:43573 #102 (3 connections now open) m31000| Wed Jan 16 17:50:31.552 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:31.966 [initandlisten] connection accepted from 127.0.0.2:41387 #104 (4 connections now open) m31002| Wed Jan 16 17:50:32.719 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1676, "optime" : { "t" : 1358376632000, "i" : 8051 }, "optimeDate" : ISODate("2013-01-16T22:50:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1665, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1665, "optime" : { "t" : 1358376631000, "i" : 1462 }, "optimeDate" : ISODate("2013-01-16T22:50:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1675, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1663, "optime" : { "t" : 1358376632000, "i" : 1489 }, "optimeDate" : ISODate("2013-01-16T22:50:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1662, "optime" : { "t" : 1358376632000, "i" : 6576 }, "optimeDate" : ISODate("2013-01-16T22:50:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1662, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1674, "optime" : { "t" : 1358376632000, "i" : 6577 }, "optimeDate" : ISODate("2013-01-16T22:50:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:33.552 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:34.784 [conn66] end connection 127.0.0.4:36512 (4 connections now open) m31000| Wed Jan 16 17:50:34.785 [initandlisten] connection accepted from 127.0.0.4:39012 #67 (5 connections now open) m31001| Wed Jan 16 17:50:35.219 [conn100] end connection 127.0.0.4:46020 (2 connections now open) m31000| Wed Jan 16 17:50:35.553 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:50:35.719 [initandlisten] connection accepted from 127.0.0.4:33853 #103 (3 connections now open) m31002| Wed Jan 16 17:50:36.723 [journal] old journal file will be removed: /data/db/testReplSet-2/journal/j._1 m31001| Wed Jan 16 17:50:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:50:37.063 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:307561 153ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1681, "optime" : { "t" : 1358376637000, "i" : 1565 }, "optimeDate" : ISODate("2013-01-16T22:50:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1670, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1670, "optime" : { "t" : 1358376636000, "i" : 8371 }, "optimeDate" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1680, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1668, "optime" : { "t" : 1358376636000, "i" : 1337 }, "optimeDate" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1667, "optime" : { "t" : 1358376636000, "i" : 6877 }, "optimeDate" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1667, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1679, "optime" : { "t" : 1358376637000, "i" : 1361 }, "optimeDate" : ISODate("2013-01-16T22:50:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:37.553 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:38.220 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:39.553 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1685, "optime" : { "t" : 1358376641000, "i" : 2511 }, "optimeDate" : ISODate("2013-01-16T22:50:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1674, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1674, "optime" : { "t" : 1358376637000, "i" : 1565 }, "optimeDate" : ISODate("2013-01-16T22:50:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1684, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1672, "optime" : { "t" : 1358376640000, "i" : 209 }, "optimeDate" : ISODate("2013-01-16T22:50:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1671, "optime" : { "t" : 1358376640000, "i" : 5502 }, "optimeDate" : ISODate("2013-01-16T22:50:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1671, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1683, "optime" : { "t" : 1358376641000, "i" : 2511 }, "optimeDate" : ISODate("2013-01-16T22:50:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:41.554 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:43.221 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:43.554 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1689, "optime" : { "t" : 1358376645000, "i" : 3817 }, "optimeDate" : ISODate("2013-01-16T22:50:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1678, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1678, "optime" : { "t" : 1358376644000, "i" : 1286 }, "optimeDate" : ISODate("2013-01-16T22:50:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1688, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1676, "optime" : { "t" : 1358376641000, "i" : 2511 }, "optimeDate" : ISODate("2013-01-16T22:50:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1675, "optime" : { "t" : 1358376644000, "i" : 4263 }, "optimeDate" : ISODate("2013-01-16T22:50:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1675, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1687, "optime" : { "t" : 1358376645000, "i" : 3817 }, "optimeDate" : ISODate("2013-01-16T22:50:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:45.555 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:50:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:50:47.555 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:48.153 [conn103] end connection 127.0.0.3:52524 (3 connections now open) m31002| Wed Jan 16 17:50:48.153 [initandlisten] connection accepted from 127.0.0.3:34583 #105 (4 connections now open) m31002| Wed Jan 16 17:50:48.222 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:49.555 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1693, "optime" : { "t" : 1358376649000, "i" : 5557 }, "optimeDate" : ISODate("2013-01-16T22:50:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1682, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1682, "optime" : { "t" : 1358376649000, "i" : 4468 }, "optimeDate" : ISODate("2013-01-16T22:50:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1692, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1680, "optime" : { "t" : 1358376645000, "i" : 3817 }, "optimeDate" : ISODate("2013-01-16T22:50:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1679, "optime" : { "t" : 1358376648000, "i" : 2857 }, "optimeDate" : ISODate("2013-01-16T22:50:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1679, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1691, "optime" : { "t" : 1358376649000, "i" : 5557 }, "optimeDate" : ISODate("2013-01-16T22:50:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:51.556 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:50:53.222 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:53.556 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1697, "optime" : { "t" : 1358376653000, "i" : 6650 }, "optimeDate" : ISODate("2013-01-16T22:50:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1686, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1686, "optime" : { "t" : 1358376649000, "i" : 5557 }, "optimeDate" : ISODate("2013-01-16T22:50:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1696, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1684, "optime" : { "t" : 1358376649000, "i" : 5557 }, "optimeDate" : ISODate("2013-01-16T22:50:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1683, "optime" : { "t" : 1358376652000, "i" : 1510 }, "optimeDate" : ISODate("2013-01-16T22:50:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1683, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1695, "optime" : { "t" : 1358376653000, "i" : 6650 }, "optimeDate" : ISODate("2013-01-16T22:50:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:50:55.556 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:57.557 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:50:57.928 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:310496 155ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1701, "optime" : { "t" : 1358376657000, "i" : 8125 }, "optimeDate" : ISODate("2013-01-16T22:50:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1690, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1690, "optime" : { "t" : 1358376656000, "i" : 1273 }, "optimeDate" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1700, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1688, "optime" : { "t" : 1358376653000, "i" : 6650 }, "optimeDate" : ISODate("2013-01-16T22:50:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:50:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1687, "optime" : { "t" : 1358376656000, "i" : 57 }, "optimeDate" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:56Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1687, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:50:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1699, "optime" : { "t" : 1358376657000, "i" : 6328 }, "optimeDate" : ISODate("2013-01-16T22:50:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:50:58.223 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:50:59.557 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:51:01.557 [conn102] end connection 127.0.0.2:43573 (2 connections now open) m31001| Wed Jan 16 17:51:01.557 [initandlisten] connection accepted from 127.0.0.2:37084 #104 (3 connections now open) m31000| Wed Jan 16 17:51:01.558 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1706, "optime" : { "t" : 1358376662000, "i" : 1755 }, "optimeDate" : ISODate("2013-01-16T22:51:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1695, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1695, "optime" : { "t" : 1358376661000, "i" : 7864 }, "optimeDate" : ISODate("2013-01-16T22:51:01Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1705, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1693, "optime" : { "t" : 1358376662000, "i" : 1424 }, "optimeDate" : ISODate("2013-01-16T22:51:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1692, "optime" : { "t" : 1358376657000, "i" : 8125 }, "optimeDate" : ISODate("2013-01-16T22:50:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1692, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1704, "optime" : { "t" : 1358376662000, "i" : 1528 }, "optimeDate" : ISODate("2013-01-16T22:51:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:51:03.224 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:03.558 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:04.789 [conn67] end connection 127.0.0.4:39012 (4 connections now open) m31000| Wed Jan 16 17:51:04.789 [initandlisten] connection accepted from 127.0.0.4:44744 #68 (5 connections now open) m31000| Wed Jan 16 17:51:05.558 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1710, "optime" : { "t" : 1358376666000, "i" : 3321 }, "optimeDate" : ISODate("2013-01-16T22:51:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1699, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1699, "optime" : { "t" : 1358376662000, "i" : 1755 }, "optimeDate" : ISODate("2013-01-16T22:51:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1709, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1697, "optime" : { "t" : 1358376666000, "i" : 1331 }, "optimeDate" : ISODate("2013-01-16T22:51:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1696, "optime" : { "t" : 1358376662000, "i" : 1755 }, "optimeDate" : ISODate("2013-01-16T22:51:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1696, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1708, "optime" : { "t" : 1358376666000, "i" : 3207 }, "optimeDate" : ISODate("2013-01-16T22:51:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:07.559 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:08.224 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:51:08.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:51:09.472 [conn104] end connection 127.0.0.2:41387 (3 connections now open) m31000| Wed Jan 16 17:51:09.559 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:09.972 [initandlisten] connection accepted from 127.0.0.2:38998 #106 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1714, "optime" : { "t" : 1358376670000, "i" : 5439 }, "optimeDate" : ISODate("2013-01-16T22:51:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1703, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1703, "optime" : { "t" : 1358376669000, "i" : 3940 }, "optimeDate" : ISODate("2013-01-16T22:51:09Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1713, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1701, "optime" : { "t" : 1358376670000, "i" : 1441 }, "optimeDate" : ISODate("2013-01-16T22:51:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1700, "optime" : { "t" : 1358376666000, "i" : 3321 }, "optimeDate" : ISODate("2013-01-16T22:51:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1700, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1712, "optime" : { "t" : 1358376670000, "i" : 5439 }, "optimeDate" : ISODate("2013-01-16T22:51:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:11.559 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:51:13.225 [conn103] end connection 127.0.0.4:33853 (2 connections now open) m31000| Wed Jan 16 17:51:13.560 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:51:13.725 [initandlisten] connection accepted from 127.0.0.4:32833 #105 (3 connections now open) m31002| Wed Jan 16 17:51:13.725 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1718, "optime" : { "t" : 1358376674000, "i" : 6726 }, "optimeDate" : ISODate("2013-01-16T22:51:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1707, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1707, "optime" : { "t" : 1358376670000, "i" : 5439 }, "optimeDate" : ISODate("2013-01-16T22:51:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1717, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1705, "optime" : { "t" : 1358376674000, "i" : 600 }, "optimeDate" : ISODate("2013-01-16T22:51:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1704, "optime" : { "t" : 1358376674000, "i" : 6350 }, "optimeDate" : ISODate("2013-01-16T22:51:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1704, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1716, "optime" : { "t" : 1358376674000, "i" : 6265 }, "optimeDate" : ISODate("2013-01-16T22:51:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:15.560 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:17.561 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:17.974 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:227981 114ms m31002| Wed Jan 16 17:51:18.158 [conn105] end connection 127.0.0.3:34583 (3 connections now open) m31002| Wed Jan 16 17:51:18.158 [initandlisten] connection accepted from 127.0.0.3:58305 #107 (4 connections now open) m31002| Wed Jan 16 17:51:18.726 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:51:18.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1722, "optime" : { "t" : 1358376678000, "i" : 8698 }, "optimeDate" : ISODate("2013-01-16T22:51:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1711, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1711, "optime" : { "t" : 1358376674000, "i" : 6726 }, "optimeDate" : ISODate("2013-01-16T22:51:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1721, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1709, "optime" : { "t" : 1358376678000, "i" : 2034 }, "optimeDate" : ISODate("2013-01-16T22:51:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1708, "optime" : { "t" : 1358376678000, "i" : 7718 }, "optimeDate" : ISODate("2013-01-16T22:51:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1708, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1720, "optime" : { "t" : 1358376678000, "i" : 8132 }, "optimeDate" : ISODate("2013-01-16T22:51:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:19.561 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:21.561 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:22.242 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:210887 105ms m31000| Wed Jan 16 17:51:22.434 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:244642 122ms STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:51:22.970 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:312297 156ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1727, "optime" : { "t" : 1358376683000, "i" : 190 }, "optimeDate" : ISODate("2013-01-16T22:51:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1716, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1716, "optime" : { "t" : 1358376682000, "i" : 4530 }, "optimeDate" : ISODate("2013-01-16T22:51:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1726, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1714, "optime" : { "t" : 1358376682000, "i" : 1168 }, "optimeDate" : ISODate("2013-01-16T22:51:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1713, "optime" : { "t" : 1358376682000, "i" : 8230 }, "optimeDate" : ISODate("2013-01-16T22:51:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1713, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1725, "optime" : { "t" : 1358376682000, "i" : 9382 }, "optimeDate" : ISODate("2013-01-16T22:51:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:23.562 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:23.727 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:25.562 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1731, "optime" : { "t" : 1358376687000, "i" : 1593 }, "optimeDate" : ISODate("2013-01-16T22:51:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1720, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1720, "optime" : { "t" : 1358376683000, "i" : 190 }, "optimeDate" : ISODate("2013-01-16T22:51:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:50:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1730, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1718, "optime" : { "t" : 1358376686000, "i" : 1154 }, "optimeDate" : ISODate("2013-01-16T22:51:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1717, "optime" : { "t" : 1358376686000, "i" : 6637 }, "optimeDate" : ISODate("2013-01-16T22:51:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1717, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1729, "optime" : { "t" : 1358376687000, "i" : 1593 }, "optimeDate" : ISODate("2013-01-16T22:51:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:27.562 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:28.727 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:29.563 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1735, "optime" : { "t" : 1358376691000, "i" : 3468 }, "optimeDate" : ISODate("2013-01-16T22:51:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1724, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1724, "optime" : { "t" : 1358376687000, "i" : 1593 }, "optimeDate" : ISODate("2013-01-16T22:51:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1734, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1722, "optime" : { "t" : 1358376687000, "i" : 1593 }, "optimeDate" : ISODate("2013-01-16T22:51:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1721, "optime" : { "t" : 1358376690000, "i" : 4679 }, "optimeDate" : ISODate("2013-01-16T22:51:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1721, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1733, "optime" : { "t" : 1358376691000, "i" : 3468 }, "optimeDate" : ISODate("2013-01-16T22:51:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:51:31.563 [conn104] end connection 127.0.0.2:37084 (2 connections now open) m31001| Wed Jan 16 17:51:31.563 [initandlisten] connection accepted from 127.0.0.2:47021 #106 (3 connections now open) m31000| Wed Jan 16 17:51:31.564 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:33.564 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:33.728 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:34.794 [conn68] end connection 127.0.0.4:44744 (4 connections now open) m31000| Wed Jan 16 17:51:34.794 [initandlisten] connection accepted from 127.0.0.4:53251 #69 (5 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:51:35.565 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1739, "optime" : { "t" : 1358376695000, "i" : 4523 }, "optimeDate" : ISODate("2013-01-16T22:51:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1728, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1728, "optime" : { "t" : 1358376694000, "i" : 5278 }, "optimeDate" : ISODate("2013-01-16T22:51:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1738, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1726, "optime" : { "t" : 1358376691000, "i" : 3468 }, "optimeDate" : ISODate("2013-01-16T22:51:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1725, "optime" : { "t" : 1358376694000, "i" : 3491 }, "optimeDate" : ISODate("2013-01-16T22:51:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1725, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1737, "optime" : { "t" : 1358376695000, "i" : 4426 }, "optimeDate" : ISODate("2013-01-16T22:51:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:37.565 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:38.728 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:39.565 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1743, "optime" : { "t" : 1358376699000, "i" : 6884 }, "optimeDate" : ISODate("2013-01-16T22:51:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1732, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1732, "optime" : { "t" : 1358376695000, "i" : 4523 }, "optimeDate" : ISODate("2013-01-16T22:51:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1742, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1730, "optime" : { "t" : 1358376695000, "i" : 4523 }, "optimeDate" : ISODate("2013-01-16T22:51:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1729, "optime" : { "t" : 1358376698000, "i" : 1629 }, "optimeDate" : ISODate("2013-01-16T22:51:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1729, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1741, "optime" : { "t" : 1358376699000, "i" : 6745 }, "optimeDate" : ISODate("2013-01-16T22:51:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:51:40.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:51:41.565 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:43.566 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:43.729 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1747, "optime" : { "t" : 1358376703000, "i" : 8034 }, "optimeDate" : ISODate("2013-01-16T22:51:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1736, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1736, "optime" : { "t" : 1358376699000, "i" : 6884 }, "optimeDate" : ISODate("2013-01-16T22:51:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1746, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1734, "optime" : { "t" : 1358376699000, "i" : 6884 }, "optimeDate" : ISODate("2013-01-16T22:51:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1733, "optime" : { "t" : 1358376702000, "i" : 51 }, "optimeDate" : ISODate("2013-01-16T22:51:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1733, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1745, "optime" : { "t" : 1358376703000, "i" : 7689 }, "optimeDate" : ISODate("2013-01-16T22:51:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:45.566 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:47.477 [conn106] end connection 127.0.0.2:38998 (3 connections now open) m31000| Wed Jan 16 17:51:47.566 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:51:47.901 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:329881 164ms m31002| Wed Jan 16 17:51:47.977 [initandlisten] connection accepted from 127.0.0.2:50995 #108 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1752, "optime" : { "t" : 1358376708000, "i" : 679 }, "optimeDate" : ISODate("2013-01-16T22:51:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1741, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1741, "optime" : { "t" : 1358376707000, "i" : 7578 }, "optimeDate" : ISODate("2013-01-16T22:51:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1751, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1739, "optime" : { "t" : 1358376703000, "i" : 8034 }, "optimeDate" : ISODate("2013-01-16T22:51:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1738, "optime" : { "t" : 1358376703000, "i" : 8034 }, "optimeDate" : ISODate("2013-01-16T22:51:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1738, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1750, "optime" : { "t" : 1358376708000, "i" : 561 }, "optimeDate" : ISODate("2013-01-16T22:51:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:51:48.164 [conn107] end connection 127.0.0.3:58305 (3 connections now open) m31002| Wed Jan 16 17:51:48.164 [initandlisten] connection accepted from 127.0.0.3:35173 #109 (4 connections now open) m31002| Wed Jan 16 17:51:48.730 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:49.567 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:51:50.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:51:51.230 [conn105] end connection 127.0.0.4:32833 (2 connections now open) m31000| Wed Jan 16 17:51:51.567 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:51:51.730 [initandlisten] connection accepted from 127.0.0.4:33962 #107 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1756, "optime" : { "t" : 1358376712000, "i" : 1405 }, "optimeDate" : ISODate("2013-01-16T22:51:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1745, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1745, "optime" : { "t" : 1358376708000, "i" : 679 }, "optimeDate" : ISODate("2013-01-16T22:51:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1755, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1743, "optime" : { "t" : 1358376708000, "i" : 679 }, "optimeDate" : ISODate("2013-01-16T22:51:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:51Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1742, "optime" : { "t" : 1358376708000, "i" : 679 }, "optimeDate" : ISODate("2013-01-16T22:51:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1742, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1754, "optime" : { "t" : 1358376712000, "i" : 1405 }, "optimeDate" : ISODate("2013-01-16T22:51:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:53.567 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:54.231 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:55.568 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1760, "optime" : { "t" : 1358376716000, "i" : 3030 }, "optimeDate" : ISODate("2013-01-16T22:51:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1749, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1749, "optime" : { "t" : 1358376715000, "i" : 2388 }, "optimeDate" : ISODate("2013-01-16T22:51:55Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1759, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1747, "optime" : { "t" : 1358376716000, "i" : 1086 }, "optimeDate" : ISODate("2013-01-16T22:51:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:51:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1746, "optime" : { "t" : 1358376712000, "i" : 1405 }, "optimeDate" : ISODate("2013-01-16T22:51:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1746, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1758, "optime" : { "t" : 1358376716000, "i" : 3030 }, "optimeDate" : ISODate("2013-01-16T22:51:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:51:57.568 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:51:59.231 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:51:59.568 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1764, "optime" : { "t" : 1358376720000, "i" : 3950 }, "optimeDate" : ISODate("2013-01-16T22:52:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1753, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1753, "optime" : { "t" : 1358376716000, "i" : 3030 }, "optimeDate" : ISODate("2013-01-16T22:51:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:51:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1763, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1751, "optime" : { "t" : 1358376720000, "i" : 1142 }, "optimeDate" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1750, "optime" : { "t" : 1358376716000, "i" : 3030 }, "optimeDate" : ISODate("2013-01-16T22:51:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1750, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:51:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1762, "optime" : { "t" : 1358376720000, "i" : 3950 }, "optimeDate" : ISODate("2013-01-16T22:52:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:52:01.569 [conn106] end connection 127.0.0.2:47021 (2 connections now open) m31001| Wed Jan 16 17:52:01.569 [initandlisten] connection accepted from 127.0.0.2:39571 #108 (3 connections now open) m31000| Wed Jan 16 17:52:01.569 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:03.569 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:04.232 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1768, "optime" : { "t" : 1358376724000, "i" : 5252 }, "optimeDate" : ISODate("2013-01-16T22:52:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1757, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1757, "optime" : { "t" : 1358376720000, "i" : 3950 }, "optimeDate" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1767, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1755, "optime" : { "t" : 1358376724000, "i" : 1605 }, "optimeDate" : ISODate("2013-01-16T22:52:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1754, "optime" : { "t" : 1358376720000, "i" : 3950 }, "optimeDate" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1754, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1766, "optime" : { "t" : 1358376724000, "i" : 4676 }, "optimeDate" : ISODate("2013-01-16T22:52:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:04.799 [conn69] end connection 127.0.0.4:53251 (4 connections now open) m31000| Wed Jan 16 17:52:04.800 [initandlisten] connection accepted from 127.0.0.4:48972 #70 (5 connections now open) m31000| Wed Jan 16 17:52:05.569 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:07.570 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:08.038 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:315892 157ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1772, "optime" : { "t" : 1358376728000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:52:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1761, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1761, "optime" : { "t" : 1358376727000, "i" : 2337 }, "optimeDate" : ISODate("2013-01-16T22:52:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1771, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1759, "optime" : { "t" : 1358376728000, "i" : 2342 }, "optimeDate" : ISODate("2013-01-16T22:52:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1758, "optime" : { "t" : 1358376724000, "i" : 5252 }, "optimeDate" : ISODate("2013-01-16T22:52:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1758, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:08Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1770, "optime" : { "t" : 1358376728000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:52:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:52:09.232 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:52:09.570 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:11.571 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1776, "optime" : { "t" : 1358376732000, "i" : 6860 }, "optimeDate" : ISODate("2013-01-16T22:52:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1765, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1765, "optime" : { "t" : 1358376728000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:52:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1775, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1763, "optime" : { "t" : 1358376732000, "i" : 1380 }, "optimeDate" : ISODate("2013-01-16T22:52:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1762, "optime" : { "t" : 1358376728000, "i" : 7249 }, "optimeDate" : ISODate("2013-01-16T22:52:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1762, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1774, "optime" : { "t" : 1358376732000, "i" : 6860 }, "optimeDate" : ISODate("2013-01-16T22:52:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:52:12.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:52:13.571 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:14.233 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:52:15.571 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1780, "optime" : { "t" : 1358376736000, "i" : 8561 }, "optimeDate" : ISODate("2013-01-16T22:52:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1769, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1769, "optime" : { "t" : 1358376732000, "i" : 6860 }, "optimeDate" : ISODate("2013-01-16T22:52:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1779, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1767, "optime" : { "t" : 1358376736000, "i" : 1766 }, "optimeDate" : ISODate("2013-01-16T22:52:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1766, "optime" : { "t" : 1358376736000, "i" : 7288 }, "optimeDate" : ISODate("2013-01-16T22:52:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1766, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1778, "optime" : { "t" : 1358376736000, "i" : 8561 }, "optimeDate" : ISODate("2013-01-16T22:52:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:17.572 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:18.169 [conn109] end connection 127.0.0.3:35173 (3 connections now open) m31002| Wed Jan 16 17:52:18.170 [initandlisten] connection accepted from 127.0.0.3:34016 #110 (4 connections now open) m31002| Wed Jan 16 17:52:19.234 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:52:19.572 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1785, "optime" : { "t" : 1358376741000, "i" : 1140 }, "optimeDate" : ISODate("2013-01-16T22:52:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1774, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1774, "optime" : { "t" : 1358376740000, "i" : 3815 }, "optimeDate" : ISODate("2013-01-16T22:52:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1784, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1772, "optime" : { "t" : 1358376740000, "i" : 1366 }, "optimeDate" : ISODate("2013-01-16T22:52:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1771, "optime" : { "t" : 1358376740000, "i" : 6910 }, "optimeDate" : ISODate("2013-01-16T22:52:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1771, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1783, "optime" : { "t" : 1358376741000, "i" : 1047 }, "optimeDate" : ISODate("2013-01-16T22:52:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:21.573 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:52:22.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:52:23.573 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:24.234 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1789, "optime" : { "t" : 1358376745000, "i" : 2454 }, "optimeDate" : ISODate("2013-01-16T22:52:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1778, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1778, "optime" : { "t" : 1358376741000, "i" : 1140 }, "optimeDate" : ISODate("2013-01-16T22:52:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1788, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1776, "optime" : { "t" : 1358376744000, "i" : 121 }, "optimeDate" : ISODate("2013-01-16T22:52:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:24Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1775, "optime" : { "t" : 1358376744000, "i" : 5788 }, "optimeDate" : ISODate("2013-01-16T22:52:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1775, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1787, "optime" : { "t" : 1358376745000, "i" : 2454 }, "optimeDate" : ISODate("2013-01-16T22:52:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:52:25.482 [conn108] end connection 127.0.0.2:50995 (3 connections now open) m31000| Wed Jan 16 17:52:25.574 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:25.982 [initandlisten] connection accepted from 127.0.0.2:50577 #111 (4 connections now open) m31000| Wed Jan 16 17:52:27.574 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:52:29.235 [conn107] end connection 127.0.0.4:33962 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1793, "optime" : { "t" : 1358376749000, "i" : 4515 }, "optimeDate" : ISODate("2013-01-16T22:52:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1782, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1782, "optime" : { "t" : 1358376748000, "i" : 1278 }, "optimeDate" : ISODate("2013-01-16T22:52:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1792, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1780, "optime" : { "t" : 1358376745000, "i" : 2454 }, "optimeDate" : ISODate("2013-01-16T22:52:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1779, "optime" : { "t" : 1358376748000, "i" : 4021 }, "optimeDate" : ISODate("2013-01-16T22:52:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1779, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1791, "optime" : { "t" : 1358376749000, "i" : 4515 }, "optimeDate" : ISODate("2013-01-16T22:52:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:29.574 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:52:29.735 [initandlisten] connection accepted from 127.0.0.4:59087 #109 (3 connections now open) m31002| Wed Jan 16 17:52:29.735 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:52:31.575 [conn108] end connection 127.0.0.2:39571 (2 connections now open) m31001| Wed Jan 16 17:52:31.575 [initandlisten] connection accepted from 127.0.0.2:54613 #110 (3 connections now open) m31000| Wed Jan 16 17:52:31.575 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:32.981 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:327375 163ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1797, "optime" : { "t" : 1358376753000, "i" : 6760 }, "optimeDate" : ISODate("2013-01-16T22:52:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1786, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1786, "optime" : { "t" : 1358376753000, "i" : 5770 }, "optimeDate" : ISODate("2013-01-16T22:52:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1796, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1784, "optime" : { "t" : 1358376749000, "i" : 4515 }, "optimeDate" : ISODate("2013-01-16T22:52:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1783, "optime" : { "t" : 1358376752000, "i" : 2705 }, "optimeDate" : ISODate("2013-01-16T22:52:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1783, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1795, "optime" : { "t" : 1358376753000, "i" : 6049 }, "optimeDate" : ISODate("2013-01-16T22:52:33Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:33.575 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:34.736 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:52:34.805 [conn70] end connection 127.0.0.4:48972 (4 connections now open) m31000| Wed Jan 16 17:52:34.805 [initandlisten] connection accepted from 127.0.0.4:60641 #71 (5 connections now open) m31000| Wed Jan 16 17:52:35.576 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:36.909 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:236937 118ms m31000| Wed Jan 16 17:52:37.576 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1801, "optime" : { "t" : 1358376757000, "i" : 6088 }, "optimeDate" : ISODate("2013-01-16T22:52:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1790, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1790, "optime" : { "t" : 1358376753000, "i" : 6760 }, "optimeDate" : ISODate("2013-01-16T22:52:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1800, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1788, "optime" : { "t" : 1358376753000, "i" : 6760 }, "optimeDate" : ISODate("2013-01-16T22:52:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1787, "optime" : { "t" : 1358376756000, "i" : 1927 }, "optimeDate" : ISODate("2013-01-16T22:52:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1787, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1799, "optime" : { "t" : 1358376757000, "i" : 5580 }, "optimeDate" : ISODate("2013-01-16T22:52:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:39.576 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:39.737 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:52:41.577 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1805, "optime" : { "t" : 1358376761000, "i" : 7082 }, "optimeDate" : ISODate("2013-01-16T22:52:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1794, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1794, "optime" : { "t" : 1358376760000, "i" : 2917 }, "optimeDate" : ISODate("2013-01-16T22:52:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1804, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1792, "optime" : { "t" : 1358376757000, "i" : 6088 }, "optimeDate" : ISODate("2013-01-16T22:52:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1791, "optime" : { "t" : 1358376760000, "i" : 1474 }, "optimeDate" : ISODate("2013-01-16T22:52:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1791, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1803, "optime" : { "t" : 1358376761000, "i" : 7039 }, "optimeDate" : ISODate("2013-01-16T22:52:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:43.577 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:44.737 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:52:44.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:52:45.577 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1809, "optime" : { "t" : 1358376765000, "i" : 8591 }, "optimeDate" : ISODate("2013-01-16T22:52:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1798, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1798, "optime" : { "t" : 1358376761000, "i" : 7082 }, "optimeDate" : ISODate("2013-01-16T22:52:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1808, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1796, "optime" : { "t" : 1358376761000, "i" : 7082 }, "optimeDate" : ISODate("2013-01-16T22:52:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1795, "optime" : { "t" : 1358376761000, "i" : 7082 }, "optimeDate" : ISODate("2013-01-16T22:52:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1795, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1807, "optime" : { "t" : 1358376765000, "i" : 8591 }, "optimeDate" : ISODate("2013-01-16T22:52:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:47.578 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:48.175 [conn110] end connection 127.0.0.3:34016 (3 connections now open) m31002| Wed Jan 16 17:52:48.175 [initandlisten] connection accepted from 127.0.0.3:34574 #112 (4 connections now open) m31000| Wed Jan 16 17:52:49.578 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:49.738 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1814, "optime" : { "t" : 1358376770000, "i" : 555 }, "optimeDate" : ISODate("2013-01-16T22:52:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1803, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1803, "optime" : { "t" : 1358376765000, "i" : 8591 }, "optimeDate" : ISODate("2013-01-16T22:52:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1813, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1801, "optime" : { "t" : 1358376765000, "i" : 8591 }, "optimeDate" : ISODate("2013-01-16T22:52:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:49Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1800, "optime" : { "t" : 1358376765000, "i" : 8591 }, "optimeDate" : ISODate("2013-01-16T22:52:45Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1800, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1812, "optime" : { "t" : 1358376770000, "i" : 555 }, "optimeDate" : ISODate("2013-01-16T22:52:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:51.579 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:53.579 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1818, "optime" : { "t" : 1358376774000, "i" : 1918 }, "optimeDate" : ISODate("2013-01-16T22:52:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1807, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1807, "optime" : { "t" : 1358376773000, "i" : 3467 }, "optimeDate" : ISODate("2013-01-16T22:52:53Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1817, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1805, "optime" : { "t" : 1358376774000, "i" : 1236 }, "optimeDate" : ISODate("2013-01-16T22:52:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1804, "optime" : { "t" : 1358376770000, "i" : 555 }, "optimeDate" : ISODate("2013-01-16T22:52:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1804, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1816, "optime" : { "t" : 1358376774000, "i" : 1918 }, "optimeDate" : ISODate("2013-01-16T22:52:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:52:54.739 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:52:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:52:55.579 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:52:57.580 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1822, "optime" : { "t" : 1358376778000, "i" : 4592 }, "optimeDate" : ISODate("2013-01-16T22:52:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1811, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1811, "optime" : { "t" : 1358376774000, "i" : 1918 }, "optimeDate" : ISODate("2013-01-16T22:52:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1821, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1809, "optime" : { "t" : 1358376778000, "i" : 2114 }, "optimeDate" : ISODate("2013-01-16T22:52:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:52:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1808, "optime" : { "t" : 1358376774000, "i" : 1918 }, "optimeDate" : ISODate("2013-01-16T22:52:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1808, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:52:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1820, "optime" : { "t" : 1358376778000, "i" : 4216 }, "optimeDate" : ISODate("2013-01-16T22:52:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:52:59.580 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:52:59.739 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:53:01.580 [conn110] end connection 127.0.0.2:54613 (2 connections now open) m31001| Wed Jan 16 17:53:01.580 [initandlisten] connection accepted from 127.0.0.2:39150 #111 (3 connections now open) m31000| Wed Jan 16 17:53:01.580 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1826, "optime" : { "t" : 1358376782000, "i" : 4222 }, "optimeDate" : ISODate("2013-01-16T22:53:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1815, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1815, "optime" : { "t" : 1358376778000, "i" : 4592 }, "optimeDate" : ISODate("2013-01-16T22:52:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:52:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1825, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1813, "optime" : { "t" : 1358376782000, "i" : 1458 }, "optimeDate" : ISODate("2013-01-16T22:53:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1812, "optime" : { "t" : 1358376778000, "i" : 4592 }, "optimeDate" : ISODate("2013-01-16T22:52:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1812, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1824, "optime" : { "t" : 1358376782000, "i" : 4222 }, "optimeDate" : ISODate("2013-01-16T22:53:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:53:03.487 [conn111] end connection 127.0.0.2:50577 (3 connections now open) m31000| Wed Jan 16 17:53:03.581 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:03.988 [initandlisten] connection accepted from 127.0.0.2:37691 #113 (4 connections now open) m31002| Wed Jan 16 17:53:04.740 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:04.810 [conn71] end connection 127.0.0.4:60641 (4 connections now open) m31000| Wed Jan 16 17:53:04.811 [initandlisten] connection accepted from 127.0.0.4:56043 #72 (5 connections now open) m31000| Wed Jan 16 17:53:05.581 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1830, "optime" : { "t" : 1358376786000, "i" : 5293 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1819, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1819, "optime" : { "t" : 1358376786000, "i" : 3963 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1829, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1817, "optime" : { "t" : 1358376786000, "i" : 1350 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1816, "optime" : { "t" : 1358376782000, "i" : 4222 }, "optimeDate" : ISODate("2013-01-16T22:53:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1816, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1828, "optime" : { "t" : 1358376786000, "i" : 5255 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:53:07.240 [conn109] end connection 127.0.0.4:59087 (2 connections now open) m31000| Wed Jan 16 17:53:07.581 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:53:07.740 [initandlisten] connection accepted from 127.0.0.4:47984 #112 (3 connections now open) m31000| Wed Jan 16 17:53:09.582 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:10.241 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1834, "optime" : { "t" : 1358376790000, "i" : 6468 }, "optimeDate" : ISODate("2013-01-16T22:53:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1823, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1823, "optime" : { "t" : 1358376786000, "i" : 5293 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1833, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1821, "optime" : { "t" : 1358376790000, "i" : 1532 }, "optimeDate" : ISODate("2013-01-16T22:53:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1820, "optime" : { "t" : 1358376786000, "i" : 5293 }, "optimeDate" : ISODate("2013-01-16T22:53:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1820, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1832, "optime" : { "t" : 1358376790000, "i" : 6468 }, "optimeDate" : ISODate("2013-01-16T22:53:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:11.582 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:13.583 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1838, "optime" : { "t" : 1358376794000, "i" : 8180 }, "optimeDate" : ISODate("2013-01-16T22:53:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1827, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1827, "optime" : { "t" : 1358376793000, "i" : 1757 }, "optimeDate" : ISODate("2013-01-16T22:53:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1837, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1825, "optime" : { "t" : 1358376794000, "i" : 1313 }, "optimeDate" : ISODate("2013-01-16T22:53:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1824, "optime" : { "t" : 1358376794000, "i" : 7662 }, "optimeDate" : ISODate("2013-01-16T22:53:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1824, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1836, "optime" : { "t" : 1358376794000, "i" : 7674 }, "optimeDate" : ISODate("2013-01-16T22:53:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:53:15.242 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:15.583 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:53:16.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:53:17.583 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:18.180 [conn112] end connection 127.0.0.3:34574 (3 connections now open) m31002| Wed Jan 16 17:53:18.180 [initandlisten] connection accepted from 127.0.0.3:50785 #114 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1843, "optime" : { "t" : 1358376799000, "i" : 307 }, "optimeDate" : ISODate("2013-01-16T22:53:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1832, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1832, "optime" : { "t" : 1358376798000, "i" : 8784 }, "optimeDate" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1842, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1830, "optime" : { "t" : 1358376798000, "i" : 1971 }, "optimeDate" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1829, "optime" : { "t" : 1358376798000, "i" : 7348 }, "optimeDate" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1829, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1841, "optime" : { "t" : 1358376799000, "i" : 307 }, "optimeDate" : ISODate("2013-01-16T22:53:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:19.584 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:20.242 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:21.584 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1847, "optime" : { "t" : 1358376803000, "i" : 2123 }, "optimeDate" : ISODate("2013-01-16T22:53:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1836, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1836, "optime" : { "t" : 1358376799000, "i" : 307 }, "optimeDate" : ISODate("2013-01-16T22:53:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1846, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1834, "optime" : { "t" : 1358376802000, "i" : 497 }, "optimeDate" : ISODate("2013-01-16T22:53:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1833, "optime" : { "t" : 1358376802000, "i" : 6308 }, "optimeDate" : ISODate("2013-01-16T22:53:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1833, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1845, "optime" : { "t" : 1358376803000, "i" : 1471 }, "optimeDate" : ISODate("2013-01-16T22:53:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:23.585 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:25.243 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:25.585 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:53:26.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1851, "optime" : { "t" : 1358376807000, "i" : 3429 }, "optimeDate" : ISODate("2013-01-16T22:53:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1840, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1840, "optime" : { "t" : 1358376806000, "i" : 2219 }, "optimeDate" : ISODate("2013-01-16T22:53:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1850, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1838, "optime" : { "t" : 1358376803000, "i" : 2123 }, "optimeDate" : ISODate("2013-01-16T22:53:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1837, "optime" : { "t" : 1358376806000, "i" : 5079 }, "optimeDate" : ISODate("2013-01-16T22:53:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1837, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1849, "optime" : { "t" : 1358376807000, "i" : 3077 }, "optimeDate" : ISODate("2013-01-16T22:53:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:27.585 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:29.586 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:30.244 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1855, "optime" : { "t" : 1358376811000, "i" : 4258 }, "optimeDate" : ISODate("2013-01-16T22:53:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1844, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1844, "optime" : { "t" : 1358376811000, "i" : 3640 }, "optimeDate" : ISODate("2013-01-16T22:53:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1854, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1842, "optime" : { "t" : 1358376807000, "i" : 3429 }, "optimeDate" : ISODate("2013-01-16T22:53:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1841, "optime" : { "t" : 1358376810000, "i" : 3977 }, "optimeDate" : ISODate("2013-01-16T22:53:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1841, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1853, "optime" : { "t" : 1358376811000, "i" : 3640 }, "optimeDate" : ISODate("2013-01-16T22:53:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:53:31.586 [conn111] end connection 127.0.0.2:39150 (2 connections now open) m31001| Wed Jan 16 17:53:31.586 [initandlisten] connection accepted from 127.0.0.2:58852 #113 (3 connections now open) m31000| Wed Jan 16 17:53:31.586 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:33.587 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:34.816 [conn72] end connection 127.0.0.4:56043 (4 connections now open) m31000| Wed Jan 16 17:53:34.816 [initandlisten] connection accepted from 127.0.0.4:51872 #73 (5 connections now open) m31002| Wed Jan 16 17:53:35.244 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:35.587 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1859, "optime" : { "t" : 1358376815000, "i" : 5681 }, "optimeDate" : ISODate("2013-01-16T22:53:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1848, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1848, "optime" : { "t" : 1358376811000, "i" : 4258 }, "optimeDate" : ISODate("2013-01-16T22:53:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1858, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1846, "optime" : { "t" : 1358376811000, "i" : 4258 }, "optimeDate" : ISODate("2013-01-16T22:53:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:35Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1845, "optime" : { "t" : 1358376814000, "i" : 2736 }, "optimeDate" : ISODate("2013-01-16T22:53:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1845, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1857, "optime" : { "t" : 1358376815000, "i" : 5463 }, "optimeDate" : ISODate("2013-01-16T22:53:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:37.588 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:39.588 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1863, "optime" : { "t" : 1358376819000, "i" : 7395 }, "optimeDate" : ISODate("2013-01-16T22:53:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1852, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1852, "optime" : { "t" : 1358376818000, "i" : 2363 }, "optimeDate" : ISODate("2013-01-16T22:53:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1862, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1850, "optime" : { "t" : 1358376815000, "i" : 5681 }, "optimeDate" : ISODate("2013-01-16T22:53:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1849, "optime" : { "t" : 1358376818000, "i" : 689 }, "optimeDate" : ISODate("2013-01-16T22:53:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1849, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1861, "optime" : { "t" : 1358376819000, "i" : 7069 }, "optimeDate" : ISODate("2013-01-16T22:53:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:53:40.245 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:53:41.493 [conn113] end connection 127.0.0.2:37691 (3 connections now open) m31000| Wed Jan 16 17:53:41.588 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:41.993 [initandlisten] connection accepted from 127.0.0.2:51703 #115 (4 connections now open) m31000| Wed Jan 16 17:53:43.588 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:43.962 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:311226 155ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1868, "optime" : { "t" : 1358376824000, "i" : 1012 }, "optimeDate" : ISODate("2013-01-16T22:53:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1857, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1857, "optime" : { "t" : 1358376819000, "i" : 7395 }, "optimeDate" : ISODate("2013-01-16T22:53:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1867, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1855, "optime" : { "t" : 1358376819000, "i" : 7395 }, "optimeDate" : ISODate("2013-01-16T22:53:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1854, "optime" : { "t" : 1358376819000, "i" : 7395 }, "optimeDate" : ISODate("2013-01-16T22:53:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1854, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1866, "optime" : { "t" : 1358376824000, "i" : 1002 }, "optimeDate" : ISODate("2013-01-16T22:53:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:53:45.245 [conn112] end connection 127.0.0.4:47984 (2 connections now open) m31000| Wed Jan 16 17:53:45.589 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:53:45.745 [initandlisten] connection accepted from 127.0.0.4:45942 #114 (3 connections now open) m31002| Wed Jan 16 17:53:45.746 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:47.589 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:48.186 [conn114] end connection 127.0.0.3:50785 (3 connections now open) m31002| Wed Jan 16 17:53:48.186 [initandlisten] connection accepted from 127.0.0.3:47594 #116 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1872, "optime" : { "t" : 1358376828000, "i" : 2193 }, "optimeDate" : ISODate("2013-01-16T22:53:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1861, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1861, "optime" : { "t" : 1358376824000, "i" : 1012 }, "optimeDate" : ISODate("2013-01-16T22:53:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1871, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1859, "optime" : { "t" : 1358376828000, "i" : 1378 }, "optimeDate" : ISODate("2013-01-16T22:53:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1858, "optime" : { "t" : 1358376824000, "i" : 1012 }, "optimeDate" : ISODate("2013-01-16T22:53:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1858, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1870, "optime" : { "t" : 1358376828000, "i" : 1379 }, "optimeDate" : ISODate("2013-01-16T22:53:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:53:48.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:53:49.590 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:50.747 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:53:51.590 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1876, "optime" : { "t" : 1358376832000, "i" : 2891 }, "optimeDate" : ISODate("2013-01-16T22:53:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1865, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1865, "optime" : { "t" : 1358376831000, "i" : 7107 }, "optimeDate" : ISODate("2013-01-16T22:53:51Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1875, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1863, "optime" : { "t" : 1358376832000, "i" : 1381 }, "optimeDate" : ISODate("2013-01-16T22:53:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1862, "optime" : { "t" : 1358376828000, "i" : 2193 }, "optimeDate" : ISODate("2013-01-16T22:53:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1862, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1874, "optime" : { "t" : 1358376832000, "i" : 2716 }, "optimeDate" : ISODate("2013-01-16T22:53:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:53.590 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:53:55.590 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:53:55.747 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1880, "optime" : { "t" : 1358376836000, "i" : 3852 }, "optimeDate" : ISODate("2013-01-16T22:53:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1869, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1869, "optime" : { "t" : 1358376832000, "i" : 2891 }, "optimeDate" : ISODate("2013-01-16T22:53:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1879, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1867, "optime" : { "t" : 1358376836000, "i" : 1504 }, "optimeDate" : ISODate("2013-01-16T22:53:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:53:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1866, "optime" : { "t" : 1358376832000, "i" : 2891 }, "optimeDate" : ISODate("2013-01-16T22:53:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1866, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1878, "optime" : { "t" : 1358376836000, "i" : 3852 }, "optimeDate" : ISODate("2013-01-16T22:53:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:53:57.591 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:53:58.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:53:59.591 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1884, "optime" : { "t" : 1358376840000, "i" : 5139 }, "optimeDate" : ISODate("2013-01-16T22:54:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1873, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1873, "optime" : { "t" : 1358376839000, "i" : 181 }, "optimeDate" : ISODate("2013-01-16T22:53:59Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1883, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1871, "optime" : { "t" : 1358376840000, "i" : 1261 }, "optimeDate" : ISODate("2013-01-16T22:54:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1870, "optime" : { "t" : 1358376836000, "i" : 3852 }, "optimeDate" : ISODate("2013-01-16T22:53:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:53:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1870, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1882, "optime" : { "t" : 1358376840000, "i" : 5139 }, "optimeDate" : ISODate("2013-01-16T22:54:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:54:00.748 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:54:01.591 [conn113] end connection 127.0.0.2:58852 (2 connections now open) m31001| Wed Jan 16 17:54:01.592 [initandlisten] connection accepted from 127.0.0.2:50261 #115 (3 connections now open) m31000| Wed Jan 16 17:54:01.592 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:03.592 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:04.076 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.5, filling with zeroes... STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:54:04.822 [conn73] end connection 127.0.0.4:51872 (4 connections now open) m31000| Wed Jan 16 17:54:04.822 [initandlisten] connection accepted from 127.0.0.4:54732 #74 (5 connections now open) m31000| Wed Jan 16 17:54:05.593 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:05.748 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:07.593 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:08.070 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.5, size: 511MB, took 3.993 secs m31000| Wed Jan 16 17:54:08.070 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:3993922 3993ms m31000| Wed Jan 16 17:54:08.070 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:125383126206669 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:87 nreturned:1 reslen:117 3994ms m31000| Wed Jan 16 17:54:08.070 [conn11] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:129158504470451 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:81 nreturned:1 reslen:37 3992ms m31002| Wed Jan 16 17:54:08.071 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.5, filling with zeroes... { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1892, "optime" : { "t" : 1358376848000, "i" : 5745 }, "optimeDate" : ISODate("2013-01-16T22:54:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1881, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1881, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:53:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1891, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1879, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1878, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1878, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1890, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:09.594 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:10.749 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:11.594 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:54:13.584 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:3124359 1562ms m31000| Wed Jan 16 17:54:13.584 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:125383126206669 ntoreturn:0 keyUpdates:0 locks(micros) r:30 nreturned:5 reslen:505 1563ms m31000| Wed Jan 16 17:54:13.594 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:15.595 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:15.750 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:54:17.235 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.5, size: 511MB, took 9.163 secs m31000| Wed Jan 16 17:54:17.237 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:7163856 3581ms m31000| Wed Jan 16 17:54:17.237 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358374960000|1 } } cursorid:125383126206669 ntoreturn:0 keyUpdates:0 locks(micros) r:39 nreturned:1 reslen:117 3584ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1901, "optime" : { "t" : 1358376857000, "i" : 677 }, "optimeDate" : ISODate("2013-01-16T22:54:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1890, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1890, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1900, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1888, "optime" : { "t" : 1358376844000, "i" : 841 }, "optimeDate" : ISODate("2013-01-16T22:54:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1887, "optime" : { "t" : 1358376853000, "i" : 2739 }, "optimeDate" : ISODate("2013-01-16T22:54:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1887, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1899, "optime" : { "t" : 1358376848000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:54:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:17.595 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:18.191 [conn116] end connection 127.0.0.3:47594 (3 connections now open) m31002| Wed Jan 16 17:54:18.191 [initandlisten] connection accepted from 127.0.0.3:35501 #117 (4 connections now open) m31002| Wed Jan 16 17:54:19.498 [conn115] end connection 127.0.0.2:51703 (3 connections now open) m31000| Wed Jan 16 17:54:19.595 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:19.998 [initandlisten] connection accepted from 127.0.0.2:52728 #118 (4 connections now open) m31002| Wed Jan 16 17:54:20.750 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:54:20.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1905, "optime" : { "t" : 1358376861000, "i" : 3870 }, "optimeDate" : ISODate("2013-01-16T22:54:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1894, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1894, "optime" : { "t" : 1358376851000, "i" : 2456 }, "optimeDate" : ISODate("2013-01-16T22:54:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1904, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1892, "optime" : { "t" : 1358376851000, "i" : 4258 }, "optimeDate" : ISODate("2013-01-16T22:54:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1891, "optime" : { "t" : 1358376860000, "i" : 4770 }, "optimeDate" : ISODate("2013-01-16T22:54:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1891, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1903, "optime" : { "t" : 1358376861000, "i" : 3552 }, "optimeDate" : ISODate("2013-01-16T22:54:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:21.596 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:54:23.251 [conn114] end connection 127.0.0.4:45942 (2 connections now open) m31000| Wed Jan 16 17:54:23.596 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:54:23.751 [initandlisten] connection accepted from 127.0.0.4:47133 #116 (3 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 17:54:25.597 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1909, "optime" : { "t" : 1358376865000, "i" : 4741 }, "optimeDate" : ISODate("2013-01-16T22:54:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1898, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1898, "optime" : { "t" : 1358376864000, "i" : 5225 }, "optimeDate" : ISODate("2013-01-16T22:54:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1908, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1896, "optime" : { "t" : 1358376861000, "i" : 3870 }, "optimeDate" : ISODate("2013-01-16T22:54:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1895, "optime" : { "t" : 1358376864000, "i" : 3709 }, "optimeDate" : ISODate("2013-01-16T22:54:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1895, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1907, "optime" : { "t" : 1358376865000, "i" : 4741 }, "optimeDate" : ISODate("2013-01-16T22:54:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:54:26.252 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:27.597 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:29.597 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1913, "optime" : { "t" : 1358376869000, "i" : 7054 }, "optimeDate" : ISODate("2013-01-16T22:54:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1902, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1902, "optime" : { "t" : 1358376865000, "i" : 4741 }, "optimeDate" : ISODate("2013-01-16T22:54:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1912, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1900, "optime" : { "t" : 1358376865000, "i" : 4741 }, "optimeDate" : ISODate("2013-01-16T22:54:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1899, "optime" : { "t" : 1358376868000, "i" : 1597 }, "optimeDate" : ISODate("2013-01-16T22:54:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1899, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1911, "optime" : { "t" : 1358376869000, "i" : 7054 }, "optimeDate" : ISODate("2013-01-16T22:54:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:54:30.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:54:31.252 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:54:31.598 [conn115] end connection 127.0.0.2:50261 (2 connections now open) m31001| Wed Jan 16 17:54:31.598 [initandlisten] connection accepted from 127.0.0.2:60883 #117 (3 connections now open) m31000| Wed Jan 16 17:54:31.598 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:33.598 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1917, "optime" : { "t" : 1358376873000, "i" : 8206 }, "optimeDate" : ISODate("2013-01-16T22:54:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1906, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1906, "optime" : { "t" : 1358376869000, "i" : 7054 }, "optimeDate" : ISODate("2013-01-16T22:54:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1916, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1904, "optime" : { "t" : 1358376869000, "i" : 7054 }, "optimeDate" : ISODate("2013-01-16T22:54:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:33Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1903, "optime" : { "t" : 1358376872000, "i" : 217 }, "optimeDate" : ISODate("2013-01-16T22:54:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1903, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1915, "optime" : { "t" : 1358376873000, "i" : 8206 }, "optimeDate" : ISODate("2013-01-16T22:54:33Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:34.827 [conn74] end connection 127.0.0.4:54732 (4 connections now open) m31000| Wed Jan 16 17:54:34.827 [initandlisten] connection accepted from 127.0.0.4:45371 #75 (5 connections now open) m31000| Wed Jan 16 17:54:35.599 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:36.253 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:37.143 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:232648 116ms m31000| Wed Jan 16 17:54:37.599 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1921, "optime" : { "t" : 1358376877000, "i" : 8725 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1910, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1910, "optime" : { "t" : 1358376877000, "i" : 4765 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1920, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1908, "optime" : { "t" : 1358376873000, "i" : 8206 }, "optimeDate" : ISODate("2013-01-16T22:54:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1907, "optime" : { "t" : 1358376873000, "i" : 8206 }, "optimeDate" : ISODate("2013-01-16T22:54:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1907, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1919, "optime" : { "t" : 1358376877000, "i" : 8714 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:39.600 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:41.253 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:41.600 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1926, "optime" : { "t" : 1358376882000, "i" : 681 }, "optimeDate" : ISODate("2013-01-16T22:54:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1915, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1915, "optime" : { "t" : 1358376877000, "i" : 8725 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1925, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1913, "optime" : { "t" : 1358376877000, "i" : 8725 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:41Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1912, "optime" : { "t" : 1358376877000, "i" : 8725 }, "optimeDate" : ISODate("2013-01-16T22:54:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1912, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1924, "optime" : { "t" : 1358376882000, "i" : 681 }, "optimeDate" : ISODate("2013-01-16T22:54:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:43.600 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:54:45.600 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1930, "optime" : { "t" : 1358376886000, "i" : 2032 }, "optimeDate" : ISODate("2013-01-16T22:54:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1919, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1919, "optime" : { "t" : 1358376882000, "i" : 681 }, "optimeDate" : ISODate("2013-01-16T22:54:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1929, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1917, "optime" : { "t" : 1358376882000, "i" : 681 }, "optimeDate" : ISODate("2013-01-16T22:54:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1916, "optime" : { "t" : 1358376882000, "i" : 681 }, "optimeDate" : ISODate("2013-01-16T22:54:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:44Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1916, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1928, "optime" : { "t" : 1358376886000, "i" : 1563 }, "optimeDate" : ISODate("2013-01-16T22:54:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:54:46.254 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:47.601 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:48.197 [conn117] end connection 127.0.0.3:35501 (3 connections now open) m31002| Wed Jan 16 17:54:48.197 [initandlisten] connection accepted from 127.0.0.3:35044 #119 (4 connections now open) m31000| Wed Jan 16 17:54:49.557 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:244067 122ms m31000| Wed Jan 16 17:54:49.601 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1934, "optime" : { "t" : 1358376890000, "i" : 1964 }, "optimeDate" : ISODate("2013-01-16T22:54:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1923, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1923, "optime" : { "t" : 1358376889000, "i" : 7415 }, "optimeDate" : ISODate("2013-01-16T22:54:49Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1933, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1921, "optime" : { "t" : 1358376890000, "i" : 1665 }, "optimeDate" : ISODate("2013-01-16T22:54:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1920, "optime" : { "t" : 1358376886000, "i" : 2032 }, "optimeDate" : ISODate("2013-01-16T22:54:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1920, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1932, "optime" : { "t" : 1358376890000, "i" : 1838 }, "optimeDate" : ISODate("2013-01-16T22:54:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:54:51.255 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:54:51.601 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:54:52.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:54:53.602 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1938, "optime" : { "t" : 1358376894000, "i" : 2934 }, "optimeDate" : ISODate("2013-01-16T22:54:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1927, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1927, "optime" : { "t" : 1358376890000, "i" : 1964 }, "optimeDate" : ISODate("2013-01-16T22:54:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1937, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1925, "optime" : { "t" : 1358376894000, "i" : 1967 }, "optimeDate" : ISODate("2013-01-16T22:54:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1924, "optime" : { "t" : 1358376890000, "i" : 1964 }, "optimeDate" : ISODate("2013-01-16T22:54:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1924, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1936, "optime" : { "t" : 1358376894000, "i" : 2332 }, "optimeDate" : ISODate("2013-01-16T22:54:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:55.602 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:56.255 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:54:57.503 [conn118] end connection 127.0.0.2:52728 (3 connections now open) m31000| Wed Jan 16 17:54:57.602 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:54:58.003 [initandlisten] connection accepted from 127.0.0.2:35279 #120 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1942, "optime" : { "t" : 1358376898000, "i" : 3940 }, "optimeDate" : ISODate("2013-01-16T22:54:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1931, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1931, "optime" : { "t" : 1358376897000, "i" : 6049 }, "optimeDate" : ISODate("2013-01-16T22:54:57Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1941, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1929, "optime" : { "t" : 1358376898000, "i" : 1399 }, "optimeDate" : ISODate("2013-01-16T22:54:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:54:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1928, "optime" : { "t" : 1358376894000, "i" : 2934 }, "optimeDate" : ISODate("2013-01-16T22:54:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1928, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:54:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1940, "optime" : { "t" : 1358376898000, "i" : 3940 }, "optimeDate" : ISODate("2013-01-16T22:54:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:54:59.603 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:01.256 [conn116] end connection 127.0.0.4:47133 (2 connections now open) m31001| Wed Jan 16 17:55:01.603 [conn117] end connection 127.0.0.2:60883 (1 connection now open) m31001| Wed Jan 16 17:55:01.603 [initandlisten] connection accepted from 127.0.0.2:49046 #118 (2 connections now open) m31000| Wed Jan 16 17:55:01.603 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:01.756 [initandlisten] connection accepted from 127.0.0.4:40851 #119 (3 connections now open) m31002| Wed Jan 16 17:55:01.756 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1946, "optime" : { "t" : 1358376902000, "i" : 5367 }, "optimeDate" : ISODate("2013-01-16T22:55:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1935, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1935, "optime" : { "t" : 1358376898000, "i" : 3940 }, "optimeDate" : ISODate("2013-01-16T22:54:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1945, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1933, "optime" : { "t" : 1358376902000, "i" : 1486 }, "optimeDate" : ISODate("2013-01-16T22:55:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1932, "optime" : { "t" : 1358376898000, "i" : 3940 }, "optimeDate" : ISODate("2013-01-16T22:54:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1932, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1944, "optime" : { "t" : 1358376902000, "i" : 5367 }, "optimeDate" : ISODate("2013-01-16T22:55:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:55:02.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:55:03.604 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:04.833 [conn75] end connection 127.0.0.4:45371 (4 connections now open) m31000| Wed Jan 16 17:55:04.833 [initandlisten] connection accepted from 127.0.0.4:48810 #76 (5 connections now open) m31000| Wed Jan 16 17:55:05.604 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:06.757 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1950, "optime" : { "t" : 1358376906000, "i" : 7141 }, "optimeDate" : ISODate("2013-01-16T22:55:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1939, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1939, "optime" : { "t" : 1358376902000, "i" : 5367 }, "optimeDate" : ISODate("2013-01-16T22:55:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1949, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1937, "optime" : { "t" : 1358376906000, "i" : 1569 }, "optimeDate" : ISODate("2013-01-16T22:55:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1936, "optime" : { "t" : 1358376906000, "i" : 6849 }, "optimeDate" : ISODate("2013-01-16T22:55:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1936, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1948, "optime" : { "t" : 1358376906000, "i" : 7141 }, "optimeDate" : ISODate("2013-01-16T22:55:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:07.605 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:09.605 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1955, "optime" : { "t" : 1358376911000, "i" : 19 }, "optimeDate" : ISODate("2013-01-16T22:55:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1944, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1944, "optime" : { "t" : 1358376910000, "i" : 4513 }, "optimeDate" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:54:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1954, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1942, "optime" : { "t" : 1358376910000, "i" : 1631 }, "optimeDate" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1941, "optime" : { "t" : 1358376910000, "i" : 7214 }, "optimeDate" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1941, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1953, "optime" : { "t" : 1358376910000, "i" : 8622 }, "optimeDate" : ISODate("2013-01-16T22:55:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:11.605 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:11.758 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:13.606 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1959, "optime" : { "t" : 1358376915000, "i" : 1125 }, "optimeDate" : ISODate("2013-01-16T22:55:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1948, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1948, "optime" : { "t" : 1358376911000, "i" : 19 }, "optimeDate" : ISODate("2013-01-16T22:55:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1958, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1946, "optime" : { "t" : 1358376914000, "i" : 1723 }, "optimeDate" : ISODate("2013-01-16T22:55:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1945, "optime" : { "t" : 1358376914000, "i" : 7440 }, "optimeDate" : ISODate("2013-01-16T22:55:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1945, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1957, "optime" : { "t" : 1358376915000, "i" : 484 }, "optimeDate" : ISODate("2013-01-16T22:55:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:15.606 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:16.758 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:17.607 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:18.202 [conn119] end connection 127.0.0.3:35044 (3 connections now open) m31002| Wed Jan 16 17:55:18.202 [initandlisten] connection accepted from 127.0.0.3:60545 #121 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1963, "optime" : { "t" : 1358376919000, "i" : 2280 }, "optimeDate" : ISODate("2013-01-16T22:55:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1952, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1952, "optime" : { "t" : 1358376915000, "i" : 1125 }, "optimeDate" : ISODate("2013-01-16T22:55:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1962, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1950, "optime" : { "t" : 1358376918000, "i" : 840 }, "optimeDate" : ISODate("2013-01-16T22:55:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1949, "optime" : { "t" : 1358376918000, "i" : 6409 }, "optimeDate" : ISODate("2013-01-16T22:55:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1949, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1961, "optime" : { "t" : 1358376919000, "i" : 2017 }, "optimeDate" : ISODate("2013-01-16T22:55:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:19.607 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:21.607 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:21.759 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1967, "optime" : { "t" : 1358376923000, "i" : 3790 }, "optimeDate" : ISODate("2013-01-16T22:55:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1956, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1956, "optime" : { "t" : 1358376922000, "i" : 6210 }, "optimeDate" : ISODate("2013-01-16T22:55:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1966, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1954, "optime" : { "t" : 1358376919000, "i" : 2280 }, "optimeDate" : ISODate("2013-01-16T22:55:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1953, "optime" : { "t" : 1358376922000, "i" : 4970 }, "optimeDate" : ISODate("2013-01-16T22:55:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1953, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1965, "optime" : { "t" : 1358376923000, "i" : 3790 }, "optimeDate" : ISODate("2013-01-16T22:55:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:23.608 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:55:25.608 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:26.759 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1971, "optime" : { "t" : 1358376927000, "i" : 5448 }, "optimeDate" : ISODate("2013-01-16T22:55:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1960, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1960, "optime" : { "t" : 1358376923000, "i" : 3790 }, "optimeDate" : ISODate("2013-01-16T22:55:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1970, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1958, "optime" : { "t" : 1358376923000, "i" : 3790 }, "optimeDate" : ISODate("2013-01-16T22:55:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1957, "optime" : { "t" : 1358376926000, "i" : 3002 }, "optimeDate" : ISODate("2013-01-16T22:55:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1957, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1969, "optime" : { "t" : 1358376927000, "i" : 5448 }, "optimeDate" : ISODate("2013-01-16T22:55:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:27.608 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:29.609 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:31.609 [conn118] end connection 127.0.0.2:49046 (2 connections now open) m31001| Wed Jan 16 17:55:31.609 [initandlisten] connection accepted from 127.0.0.2:55662 #120 (3 connections now open) m31000| Wed Jan 16 17:55:31.609 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1975, "optime" : { "t" : 1358376931000, "i" : 6734 }, "optimeDate" : ISODate("2013-01-16T22:55:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1964, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1964, "optime" : { "t" : 1358376927000, "i" : 5448 }, "optimeDate" : ISODate("2013-01-16T22:55:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1974, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1962, "optime" : { "t" : 1358376927000, "i" : 5448 }, "optimeDate" : ISODate("2013-01-16T22:55:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 m31002| Wed Jan 16 17:55:31.760 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1961, "optime" : { "t" : 1358376930000, "i" : 2099 }, "optimeDate" : ISODate("2013-01-16T22:55:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1961, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1973, "optime" : { "t" : 1358376931000, "i" : 6734 }, "optimeDate" : ISODate("2013-01-16T22:55:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:33.610 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:34.838 [conn76] end connection 127.0.0.4:48810 (4 connections now open) m31000| Wed Jan 16 17:55:34.838 [initandlisten] connection accepted from 127.0.0.4:47366 #77 (5 connections now open) m31001| Wed Jan 16 17:55:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:55:35.509 [conn120] end connection 127.0.0.2:35279 (3 connections now open) m31000| Wed Jan 16 17:55:35.610 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1979, "optime" : { "t" : 1358376935000, "i" : 8395 }, "optimeDate" : ISODate("2013-01-16T22:55:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1968, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1968, "optime" : { "t" : 1358376931000, "i" : 6734 }, "optimeDate" : ISODate("2013-01-16T22:55:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1978, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1966, "optime" : { "t" : 1358376931000, "i" : 6734 }, "optimeDate" : ISODate("2013-01-16T22:55:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1965, "optime" : { "t" : 1358376934000, "i" : 136 }, "optimeDate" : ISODate("2013-01-16T22:55:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1965, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1977, "optime" : { "t" : 1358376935000, "i" : 8336 }, "optimeDate" : ISODate("2013-01-16T22:55:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:55:36.009 [initandlisten] connection accepted from 127.0.0.2:36076 #122 (4 connections now open) m31002| Wed Jan 16 17:55:36.761 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:37.610 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:39.261 [conn119] end connection 127.0.0.4:40851 (2 connections now open) m31000| Wed Jan 16 17:55:39.611 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:55:39.761 [initandlisten] connection accepted from 127.0.0.4:40173 #121 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1984, "optime" : { "t" : 1358376940000, "i" : 126 }, "optimeDate" : ISODate("2013-01-16T22:55:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1973, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1973, "optime" : { "t" : 1358376935000, "i" : 8395 }, "optimeDate" : ISODate("2013-01-16T22:55:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1983, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1971, "optime" : { "t" : 1358376935000, "i" : 8395 }, "optimeDate" : ISODate("2013-01-16T22:55:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:39Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1970, "optime" : { "t" : 1358376935000, "i" : 8395 }, "optimeDate" : ISODate("2013-01-16T22:55:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1970, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1982, "optime" : { "t" : 1358376940000, "i" : 126 }, "optimeDate" : ISODate("2013-01-16T22:55:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:41.611 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:42.262 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:43.611 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1988, "optime" : { "t" : 1358376944000, "i" : 1720 }, "optimeDate" : ISODate("2013-01-16T22:55:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1977, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1977, "optime" : { "t" : 1358376943000, "i" : 3009 }, "optimeDate" : ISODate("2013-01-16T22:55:43Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1987, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1975, "optime" : { "t" : 1358376940000, "i" : 126 }, "optimeDate" : ISODate("2013-01-16T22:55:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1974, "optime" : { "t" : 1358376940000, "i" : 126 }, "optimeDate" : ISODate("2013-01-16T22:55:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1974, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1986, "optime" : { "t" : 1358376944000, "i" : 1720 }, "optimeDate" : ISODate("2013-01-16T22:55:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:45.612 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:47.262 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:47.612 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:48.207 [conn121] end connection 127.0.0.3:60545 (3 connections now open) m31002| Wed Jan 16 17:55:48.207 [initandlisten] connection accepted from 127.0.0.3:57252 #123 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1992, "optime" : { "t" : 1358376948000, "i" : 3397 }, "optimeDate" : ISODate("2013-01-16T22:55:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1981, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1981, "optime" : { "t" : 1358376944000, "i" : 1720 }, "optimeDate" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1991, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1979, "optime" : { "t" : 1358376948000, "i" : 1561 }, "optimeDate" : ISODate("2013-01-16T22:55:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1978, "optime" : { "t" : 1358376944000, "i" : 1720 }, "optimeDate" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1978, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1990, "optime" : { "t" : 1358376948000, "i" : 3397 }, "optimeDate" : ISODate("2013-01-16T22:55:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:49.613 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:51.613 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:55:52.263 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1996, "optime" : { "t" : 1358376952000, "i" : 5209 }, "optimeDate" : ISODate("2013-01-16T22:55:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1985, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1985, "optime" : { "t" : 1358376948000, "i" : 3397 }, "optimeDate" : ISODate("2013-01-16T22:55:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1995, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1983, "optime" : { "t" : 1358376952000, "i" : 1261 }, "optimeDate" : ISODate("2013-01-16T22:55:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1982, "optime" : { "t" : 1358376948000, "i" : 3397 }, "optimeDate" : ISODate("2013-01-16T22:55:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1982, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1994, "optime" : { "t" : 1358376952000, "i" : 5209 }, "optimeDate" : ISODate("2013-01-16T22:55:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:55:53.613 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:55.613 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2000, "optime" : { "t" : 1358376956000, "i" : 6121 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1989, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1989, "optime" : { "t" : 1358376956000, "i" : 73 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1999, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1987, "optime" : { "t" : 1358376956000, "i" : 1953 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:55:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1986, "optime" : { "t" : 1358376952000, "i" : 5209 }, "optimeDate" : ISODate("2013-01-16T22:55:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:56Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1986, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1998, "optime" : { "t" : 1358376956000, "i" : 6121 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:55:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:55:57.264 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:55:57.614 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:55:59.614 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2004, "optime" : { "t" : 1358376960000, "i" : 7354 }, "optimeDate" : ISODate("2013-01-16T22:56:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1993, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1993, "optime" : { "t" : 1358376956000, "i" : 6121 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:55:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2003, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1991, "optime" : { "t" : 1358376960000, "i" : 1329 }, "optimeDate" : ISODate("2013-01-16T22:56:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1990, "optime" : { "t" : 1358376956000, "i" : 6121 }, "optimeDate" : ISODate("2013-01-16T22:55:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1990, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2002, "optime" : { "t" : 1358376960000, "i" : 6648 }, "optimeDate" : ISODate("2013-01-16T22:56:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:56:01.615 [conn120] end connection 127.0.0.2:55662 (2 connections now open) m31001| Wed Jan 16 17:56:01.615 [initandlisten] connection accepted from 127.0.0.2:54250 #122 (3 connections now open) m31000| Wed Jan 16 17:56:01.615 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:02.264 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:03.615 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:04.578 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:328052 164ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2008, "optime" : { "t" : 1358376964000, "i" : 8439 }, "optimeDate" : ISODate("2013-01-16T22:56:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1997, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1997, "optime" : { "t" : 1358376960000, "i" : 7354 }, "optimeDate" : ISODate("2013-01-16T22:56:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2007, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1995, "optime" : { "t" : 1358376964000, "i" : 2230 }, "optimeDate" : ISODate("2013-01-16T22:56:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1994, "optime" : { "t" : 1358376960000, "i" : 7354 }, "optimeDate" : ISODate("2013-01-16T22:56:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1994, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2006, "optime" : { "t" : 1358376964000, "i" : 8439 }, "optimeDate" : ISODate("2013-01-16T22:56:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:04.843 [conn77] end connection 127.0.0.4:47366 (4 connections now open) m31000| Wed Jan 16 17:56:04.843 [initandlisten] connection accepted from 127.0.0.4:42269 #78 (5 connections now open) m31000| Wed Jan 16 17:56:05.616 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:56:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 17:56:07.265 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:07.616 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2012, "optime" : { "t" : 1358376968000, "i" : 8785 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2001, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2001, "optime" : { "t" : 1358376968000, "i" : 4590 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2011, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1999, "optime" : { "t" : 1358376968000, "i" : 1354 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1998, "optime" : { "t" : 1358376968000, "i" : 7511 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1998, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:08Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2010, "optime" : { "t" : 1358376968000, "i" : 8785 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:09.617 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:11.617 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:12.265 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2017, "optime" : { "t" : 1358376973000, "i" : 1746 }, "optimeDate" : ISODate("2013-01-16T22:56:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2006, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2006, "optime" : { "t" : 1358376968000, "i" : 8785 }, "optimeDate" : ISODate("2013-01-16T22:56:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:55:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2016, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2004, "optime" : { "t" : 1358376972000, "i" : 1505 }, "optimeDate" : ISODate("2013-01-16T22:56:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:12Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2003, "optime" : { "t" : 1358376972000, "i" : 6771 }, "optimeDate" : ISODate("2013-01-16T22:56:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2003, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2015, "optime" : { "t" : 1358376973000, "i" : 1746 }, "optimeDate" : ISODate("2013-01-16T22:56:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:56:13.514 [conn122] end connection 127.0.0.2:36076 (3 connections now open) m31000| Wed Jan 16 17:56:13.617 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:14.014 [initandlisten] connection accepted from 127.0.0.2:56133 #124 (4 connections now open) m31000| Wed Jan 16 17:56:15.618 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:56:17.266 [conn121] end connection 127.0.0.4:40173 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2021, "optime" : { "t" : 1358376977000, "i" : 2744 }, "optimeDate" : ISODate("2013-01-16T22:56:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2010, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2010, "optime" : { "t" : 1358376976000, "i" : 2709 }, "optimeDate" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2020, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2008, "optime" : { "t" : 1358376976000, "i" : 221 }, "optimeDate" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2007, "optime" : { "t" : 1358376976000, "i" : 5790 }, "optimeDate" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2007, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2019, "optime" : { "t" : 1358376977000, "i" : 2386 }, "optimeDate" : ISODate("2013-01-16T22:56:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:17.618 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:56:17.766 [initandlisten] connection accepted from 127.0.0.4:60794 #123 (3 connections now open) m31002| Wed Jan 16 17:56:17.766 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:56:18.212 [conn123] end connection 127.0.0.3:57252 (3 connections now open) m31002| Wed Jan 16 17:56:18.212 [initandlisten] connection accepted from 127.0.0.3:38928 #125 (4 connections now open) m31000| Wed Jan 16 17:56:19.619 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2025, "optime" : { "t" : 1358376981000, "i" : 4250 }, "optimeDate" : ISODate("2013-01-16T22:56:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2014, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2014, "optime" : { "t" : 1358376977000, "i" : 2744 }, "optimeDate" : ISODate("2013-01-16T22:56:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2024, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2012, "optime" : { "t" : 1358376977000, "i" : 2744 }, "optimeDate" : ISODate("2013-01-16T22:56:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2011, "optime" : { "t" : 1358376980000, "i" : 4474 }, "optimeDate" : ISODate("2013-01-16T22:56:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2011, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2023, "optime" : { "t" : 1358376981000, "i" : 4250 }, "optimeDate" : ISODate("2013-01-16T22:56:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:21.619 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:22.767 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:23.619 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2029, "optime" : { "t" : 1358376985000, "i" : 4878 }, "optimeDate" : ISODate("2013-01-16T22:56:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2018, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2018, "optime" : { "t" : 1358376981000, "i" : 4250 }, "optimeDate" : ISODate("2013-01-16T22:56:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2028, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2016, "optime" : { "t" : 1358376981000, "i" : 4250 }, "optimeDate" : ISODate("2013-01-16T22:56:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2015, "optime" : { "t" : 1358376984000, "i" : 3231 }, "optimeDate" : ISODate("2013-01-16T22:56:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2015, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2027, "optime" : { "t" : 1358376985000, "i" : 4878 }, "optimeDate" : ISODate("2013-01-16T22:56:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:25.620 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:27.620 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:27.768 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:56:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:56:29.620 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2033, "optime" : { "t" : 1358376989000, "i" : 6444 }, "optimeDate" : ISODate("2013-01-16T22:56:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2022, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2022, "optime" : { "t" : 1358376989000, "i" : 137 }, "optimeDate" : ISODate("2013-01-16T22:56:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2032, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2020, "optime" : { "t" : 1358376985000, "i" : 4878 }, "optimeDate" : ISODate("2013-01-16T22:56:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2019, "optime" : { "t" : 1358376988000, "i" : 2187 }, "optimeDate" : ISODate("2013-01-16T22:56:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2019, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2031, "optime" : { "t" : 1358376989000, "i" : 6134 }, "optimeDate" : ISODate("2013-01-16T22:56:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:56:31.621 [conn122] end connection 127.0.0.2:54250 (2 connections now open) m31001| Wed Jan 16 17:56:31.621 [initandlisten] connection accepted from 127.0.0.2:38549 #124 (3 connections now open) m31000| Wed Jan 16 17:56:31.621 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:32.768 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:33.621 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2037, "optime" : { "t" : 1358376993000, "i" : 7049 }, "optimeDate" : ISODate("2013-01-16T22:56:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2026, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2026, "optime" : { "t" : 1358376989000, "i" : 6444 }, "optimeDate" : ISODate("2013-01-16T22:56:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2036, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2024, "optime" : { "t" : 1358376989000, "i" : 6444 }, "optimeDate" : ISODate("2013-01-16T22:56:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2023, "optime" : { "t" : 1358376992000, "i" : 1605 }, "optimeDate" : ISODate("2013-01-16T22:56:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2023, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2035, "optime" : { "t" : 1358376993000, "i" : 7049 }, "optimeDate" : ISODate("2013-01-16T22:56:33Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:34.848 [conn78] end connection 127.0.0.4:42269 (4 connections now open) m31000| Wed Jan 16 17:56:34.848 [initandlisten] connection accepted from 127.0.0.4:60623 #79 (5 connections now open) m31000| Wed Jan 16 17:56:35.622 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:37.622 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:37.769 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2041, "optime" : { "t" : 1358376997000, "i" : 8021 }, "optimeDate" : ISODate("2013-01-16T22:56:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2030, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2030, "optime" : { "t" : 1358376993000, "i" : 7049 }, "optimeDate" : ISODate("2013-01-16T22:56:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2040, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2028, "optime" : { "t" : 1358376993000, "i" : 7049 }, "optimeDate" : ISODate("2013-01-16T22:56:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2027, "optime" : { "t" : 1358376996000, "i" : 338 }, "optimeDate" : ISODate("2013-01-16T22:56:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2027, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2039, "optime" : { "t" : 1358376997000, "i" : 7756 }, "optimeDate" : ISODate("2013-01-16T22:56:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:56:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:56:39.622 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:41.623 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2046, "optime" : { "t" : 1358377002000, "i" : 416 }, "optimeDate" : ISODate("2013-01-16T22:56:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2035, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2035, "optime" : { "t" : 1358377001000, "i" : 4916 }, "optimeDate" : ISODate("2013-01-16T22:56:41Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2045, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2033, "optime" : { "t" : 1358376997000, "i" : 8021 }, "optimeDate" : ISODate("2013-01-16T22:56:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2032, "optime" : { "t" : 1358376997000, "i" : 8021 }, "optimeDate" : ISODate("2013-01-16T22:56:37Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2032, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2044, "optime" : { "t" : 1358377002000, "i" : 416 }, "optimeDate" : ISODate("2013-01-16T22:56:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:56:42.769 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:43.623 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:56:45.623 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2050, "optime" : { "t" : 1358377006000, "i" : 1410 }, "optimeDate" : ISODate("2013-01-16T22:56:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2039, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2039, "optime" : { "t" : 1358377002000, "i" : 416 }, "optimeDate" : ISODate("2013-01-16T22:56:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2049, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2037, "optime" : { "t" : 1358377002000, "i" : 416 }, "optimeDate" : ISODate("2013-01-16T22:56:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:45Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2036, "optime" : { "t" : 1358377002000, "i" : 416 }, "optimeDate" : ISODate("2013-01-16T22:56:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2036, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2048, "optime" : { "t" : 1358377006000, "i" : 1410 }, "optimeDate" : ISODate("2013-01-16T22:56:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:47.624 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:47.770 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:56:48.218 [conn125] end connection 127.0.0.3:38928 (3 connections now open) m31002| Wed Jan 16 17:56:48.218 [initandlisten] connection accepted from 127.0.0.3:54677 #126 (4 connections now open) m31000| Wed Jan 16 17:56:49.624 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2054, "optime" : { "t" : 1358377010000, "i" : 2893 }, "optimeDate" : ISODate("2013-01-16T22:56:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2043, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2043, "optime" : { "t" : 1358377006000, "i" : 1410 }, "optimeDate" : ISODate("2013-01-16T22:56:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2053, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2041, "optime" : { "t" : 1358377010000, "i" : 1683 }, "optimeDate" : ISODate("2013-01-16T22:56:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2040, "optime" : { "t" : 1358377006000, "i" : 1410 }, "optimeDate" : ISODate("2013-01-16T22:56:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2040, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2052, "optime" : { "t" : 1358377010000, "i" : 2893 }, "optimeDate" : ISODate("2013-01-16T22:56:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:56:51.519 [conn124] end connection 127.0.0.2:56133 (3 connections now open) m31000| Wed Jan 16 17:56:51.624 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:52.019 [initandlisten] connection accepted from 127.0.0.2:40431 #127 (4 connections now open) m31002| Wed Jan 16 17:56:52.771 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:56:53.625 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2058, "optime" : { "t" : 1358377014000, "i" : 4353 }, "optimeDate" : ISODate("2013-01-16T22:56:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2047, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2047, "optime" : { "t" : 1358377010000, "i" : 2893 }, "optimeDate" : ISODate("2013-01-16T22:56:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2057, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2045, "optime" : { "t" : 1358377014000, "i" : 1980 }, "optimeDate" : ISODate("2013-01-16T22:56:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2044, "optime" : { "t" : 1358377010000, "i" : 2893 }, "optimeDate" : ISODate("2013-01-16T22:56:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2044, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2056, "optime" : { "t" : 1358377014000, "i" : 3439 }, "optimeDate" : ISODate("2013-01-16T22:56:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:56:55.271 [conn123] end connection 127.0.0.4:60794 (2 connections now open) m31000| Wed Jan 16 17:56:55.625 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:56:55.771 [initandlisten] connection accepted from 127.0.0.4:41974 #125 (3 connections now open) m31000| Wed Jan 16 17:56:57.625 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:56:58.272 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2062, "optime" : { "t" : 1358377018000, "i" : 5260 }, "optimeDate" : ISODate("2013-01-16T22:56:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2051, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2051, "optime" : { "t" : 1358377014000, "i" : 4353 }, "optimeDate" : ISODate("2013-01-16T22:56:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2061, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2049, "optime" : { "t" : 1358377018000, "i" : 2271 }, "optimeDate" : ISODate("2013-01-16T22:56:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:56:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2048, "optime" : { "t" : 1358377014000, "i" : 4353 }, "optimeDate" : ISODate("2013-01-16T22:56:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:56:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2048, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2060, "optime" : { "t" : 1358377018000, "i" : 5260 }, "optimeDate" : ISODate("2013-01-16T22:56:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:56:59.626 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:57:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:57:01.626 [conn124] end connection 127.0.0.2:38549 (2 connections now open) m31001| Wed Jan 16 17:57:01.626 [initandlisten] connection accepted from 127.0.0.2:55620 #126 (3 connections now open) m31000| Wed Jan 16 17:57:01.626 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2066, "optime" : { "t" : 1358377022000, "i" : 5625 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2055, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2055, "optime" : { "t" : 1358377022000, "i" : 142 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2065, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2053, "optime" : { "t" : 1358377022000, "i" : 1799 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2052, "optime" : { "t" : 1358377018000, "i" : 5260 }, "optimeDate" : ISODate("2013-01-16T22:56:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2052, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2064, "optime" : { "t" : 1358377022000, "i" : 5216 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:57:03.272 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:03.627 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:04.854 [conn79] end connection 127.0.0.4:60623 (4 connections now open) m31000| Wed Jan 16 17:57:04.854 [initandlisten] connection accepted from 127.0.0.4:53779 #80 (5 connections now open) m31000| Wed Jan 16 17:57:05.627 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2070, "optime" : { "t" : 1358377026000, "i" : 7026 }, "optimeDate" : ISODate("2013-01-16T22:57:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2059, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2059, "optime" : { "t" : 1358377022000, "i" : 5625 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2069, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2057, "optime" : { "t" : 1358377026000, "i" : 1685 }, "optimeDate" : ISODate("2013-01-16T22:57:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2056, "optime" : { "t" : 1358377022000, "i" : 5625 }, "optimeDate" : ISODate("2013-01-16T22:57:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2056, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2068, "optime" : { "t" : 1358377026000, "i" : 7017 }, "optimeDate" : ISODate("2013-01-16T22:57:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:07.627 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:08.273 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:09.628 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:57:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2074, "optime" : { "t" : 1358377030000, "i" : 8259 }, "optimeDate" : ISODate("2013-01-16T22:57:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2063, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2063, "optime" : { "t" : 1358377026000, "i" : 7026 }, "optimeDate" : ISODate("2013-01-16T22:57:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2073, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2061, "optime" : { "t" : 1358377030000, "i" : 1480 }, "optimeDate" : ISODate("2013-01-16T22:57:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2060, "optime" : { "t" : 1358377030000, "i" : 7011 }, "optimeDate" : ISODate("2013-01-16T22:57:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2060, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2072, "optime" : { "t" : 1358377030000, "i" : 8259 }, "optimeDate" : ISODate("2013-01-16T22:57:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:11.628 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:13.274 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:13.629 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:14.542 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:410643 205ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2079, "optime" : { "t" : 1358377035000, "i" : 40 }, "optimeDate" : ISODate("2013-01-16T22:57:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2068, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2068, "optime" : { "t" : 1358377034000, "i" : 2915 }, "optimeDate" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2078, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2066, "optime" : { "t" : 1358377034000, "i" : 1724 }, "optimeDate" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2065, "optime" : { "t" : 1358377034000, "i" : 8551 }, "optimeDate" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2065, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2077, "optime" : { "t" : 1358377035000, "i" : 40 }, "optimeDate" : ISODate("2013-01-16T22:57:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:15.629 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:17.629 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:18.223 [conn126] end connection 127.0.0.3:54677 (3 connections now open) m31002| Wed Jan 16 17:57:18.223 [initandlisten] connection accepted from 127.0.0.3:53238 #128 (4 connections now open) m31002| Wed Jan 16 17:57:18.274 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2083, "optime" : { "t" : 1358377039000, "i" : 1238 }, "optimeDate" : ISODate("2013-01-16T22:57:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2072, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2072, "optime" : { "t" : 1358377035000, "i" : 40 }, "optimeDate" : ISODate("2013-01-16T22:57:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2082, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2070, "optime" : { "t" : 1358377038000, "i" : 1570 }, "optimeDate" : ISODate("2013-01-16T22:57:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2069, "optime" : { "t" : 1358377038000, "i" : 7493 }, "optimeDate" : ISODate("2013-01-16T22:57:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2069, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2081, "optime" : { "t" : 1358377039000, "i" : 730 }, "optimeDate" : ISODate("2013-01-16T22:57:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:19.630 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:21.630 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:23.275 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2087, "optime" : { "t" : 1358377043000, "i" : 3178 }, "optimeDate" : ISODate("2013-01-16T22:57:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2076, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2076, "optime" : { "t" : 1358377039000, "i" : 1238 }, "optimeDate" : ISODate("2013-01-16T22:57:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2086, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2074, "optime" : { "t" : 1358377042000, "i" : 653 }, "optimeDate" : ISODate("2013-01-16T22:57:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:23Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2073, "optime" : { "t" : 1358377042000, "i" : 5938 }, "optimeDate" : ISODate("2013-01-16T22:57:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2073, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2085, "optime" : { "t" : 1358377043000, "i" : 3178 }, "optimeDate" : ISODate("2013-01-16T22:57:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:23.631 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:25.631 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2091, "optime" : { "t" : 1358377047000, "i" : 4703 }, "optimeDate" : ISODate("2013-01-16T22:57:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2080, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2080, "optime" : { "t" : 1358377047000, "i" : 244 }, "optimeDate" : ISODate("2013-01-16T22:57:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2090, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2078, "optime" : { "t" : 1358377043000, "i" : 3178 }, "optimeDate" : ISODate("2013-01-16T22:57:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2077, "optime" : { "t" : 1358377046000, "i" : 4185 }, "optimeDate" : ISODate("2013-01-16T22:57:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2077, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2089, "optime" : { "t" : 1358377047000, "i" : 4703 }, "optimeDate" : ISODate("2013-01-16T22:57:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:27.631 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:28.275 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:57:29.525 [conn127] end connection 127.0.0.2:40431 (3 connections now open) m31000| Wed Jan 16 17:57:29.632 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:30.025 [initandlisten] connection accepted from 127.0.0.2:58069 #129 (4 connections now open) m31001| Wed Jan 16 17:57:31.632 [conn126] end connection 127.0.0.2:55620 (2 connections now open) m31001| Wed Jan 16 17:57:31.632 [initandlisten] connection accepted from 127.0.0.2:53354 #127 (3 connections now open) m31000| Wed Jan 16 17:57:31.632 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2095, "optime" : { "t" : 1358377051000, "i" : 6046 }, "optimeDate" : ISODate("2013-01-16T22:57:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2084, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2084, "optime" : { "t" : 1358377047000, "i" : 4703 }, "optimeDate" : ISODate("2013-01-16T22:57:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2094, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2082, "optime" : { "t" : 1358377047000, "i" : 4703 }, "optimeDate" : ISODate("2013-01-16T22:57:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2081, "optime" : { "t" : 1358377050000, "i" : 2796 }, "optimeDate" : ISODate("2013-01-16T22:57:30Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2081, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2093, "optime" : { "t" : 1358377051000, "i" : 6046 }, "optimeDate" : ISODate("2013-01-16T22:57:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:57:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 17:57:33.276 [conn125] end connection 127.0.0.4:41974 (2 connections now open) m31000| Wed Jan 16 17:57:33.633 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:57:33.776 [initandlisten] connection accepted from 127.0.0.4:45544 #128 (3 connections now open) m31002| Wed Jan 16 17:57:33.776 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:34.859 [conn80] end connection 127.0.0.4:53779 (4 connections now open) m31000| Wed Jan 16 17:57:34.859 [initandlisten] connection accepted from 127.0.0.4:59951 #81 (5 connections now open) m31000| Wed Jan 16 17:57:35.633 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2099, "optime" : { "t" : 1358377055000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T22:57:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2088, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2088, "optime" : { "t" : 1358377054000, "i" : 1775 }, "optimeDate" : ISODate("2013-01-16T22:57:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2098, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2086, "optime" : { "t" : 1358377051000, "i" : 6046 }, "optimeDate" : ISODate("2013-01-16T22:57:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2085, "optime" : { "t" : 1358377054000, "i" : 410 }, "optimeDate" : ISODate("2013-01-16T22:57:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2085, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2097, "optime" : { "t" : 1358377055000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T22:57:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:37.633 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:38.777 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:39.633 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2103, "optime" : { "t" : 1358377059000, "i" : 8833 }, "optimeDate" : ISODate("2013-01-16T22:57:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2092, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2092, "optime" : { "t" : 1358377055000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T22:57:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2102, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2090, "optime" : { "t" : 1358377055000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T22:57:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2089, "optime" : { "t" : 1358377055000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T22:57:35Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2089, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2101, "optime" : { "t" : 1358377059000, "i" : 8833 }, "optimeDate" : ISODate("2013-01-16T22:57:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:41.634 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:57:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:57:43.634 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:43.778 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2108, "optime" : { "t" : 1358377064000, "i" : 991 }, "optimeDate" : ISODate("2013-01-16T22:57:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2097, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2097, "optime" : { "t" : 1358377059000, "i" : 8833 }, "optimeDate" : ISODate("2013-01-16T22:57:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2107, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2095, "optime" : { "t" : 1358377059000, "i" : 8833 }, "optimeDate" : ISODate("2013-01-16T22:57:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2094, "optime" : { "t" : 1358377059000, "i" : 8833 }, "optimeDate" : ISODate("2013-01-16T22:57:39Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2094, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2106, "optime" : { "t" : 1358377064000, "i" : 989 }, "optimeDate" : ISODate("2013-01-16T22:57:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:45.634 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:47.635 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:48.228 [conn128] end connection 127.0.0.3:53238 (3 connections now open) m31002| Wed Jan 16 17:57:48.228 [initandlisten] connection accepted from 127.0.0.3:42725 #130 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2112, "optime" : { "t" : 1358377068000, "i" : 2602 }, "optimeDate" : ISODate("2013-01-16T22:57:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2101, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2101, "optime" : { "t" : 1358377067000, "i" : 3052 }, "optimeDate" : ISODate("2013-01-16T22:57:47Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2111, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2099, "optime" : { "t" : 1358377068000, "i" : 2015 }, "optimeDate" : ISODate("2013-01-16T22:57:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2098, "optime" : { "t" : 1358377064000, "i" : 991 }, "optimeDate" : ISODate("2013-01-16T22:57:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2098, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2110, "optime" : { "t" : 1358377068000, "i" : 2602 }, "optimeDate" : ISODate("2013-01-16T22:57:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:57:48.778 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:49.635 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:57:51.635 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2116, "optime" : { "t" : 1358377072000, "i" : 4037 }, "optimeDate" : ISODate("2013-01-16T22:57:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2105, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2105, "optime" : { "t" : 1358377068000, "i" : 2602 }, "optimeDate" : ISODate("2013-01-16T22:57:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2115, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2103, "optime" : { "t" : 1358377072000, "i" : 2572 }, "optimeDate" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2102, "optime" : { "t" : 1358377068000, "i" : 2602 }, "optimeDate" : ISODate("2013-01-16T22:57:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2102, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2114, "optime" : { "t" : 1358377072000, "i" : 4037 }, "optimeDate" : ISODate("2013-01-16T22:57:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:53.636 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:53.779 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:55.636 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2120, "optime" : { "t" : 1358377076000, "i" : 5263 }, "optimeDate" : ISODate("2013-01-16T22:57:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2109, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2109, "optime" : { "t" : 1358377072000, "i" : 4037 }, "optimeDate" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2119, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2107, "optime" : { "t" : 1358377076000, "i" : 1563 }, "optimeDate" : ISODate("2013-01-16T22:57:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:57:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2106, "optime" : { "t" : 1358377072000, "i" : 4037 }, "optimeDate" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2106, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2118, "optime" : { "t" : 1358377076000, "i" : 4972 }, "optimeDate" : ISODate("2013-01-16T22:57:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:57:57.637 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:57:58.780 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:57:59.637 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2124, "optime" : { "t" : 1358377080000, "i" : 7014 }, "optimeDate" : ISODate("2013-01-16T22:58:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2113, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2113, "optime" : { "t" : 1358377080000, "i" : 169 }, "optimeDate" : ISODate("2013-01-16T22:58:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:57:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2123, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2111, "optime" : { "t" : 1358377080000, "i" : 1815 }, "optimeDate" : ISODate("2013-01-16T22:58:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2110, "optime" : { "t" : 1358377076000, "i" : 5263 }, "optimeDate" : ISODate("2013-01-16T22:57:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2110, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2122, "optime" : { "t" : 1358377080000, "i" : 7014 }, "optimeDate" : ISODate("2013-01-16T22:58:00Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:58:01.637 [conn127] end connection 127.0.0.2:53354 (2 connections now open) m31001| Wed Jan 16 17:58:01.637 [initandlisten] connection accepted from 127.0.0.2:45980 #129 (3 connections now open) m31000| Wed Jan 16 17:58:01.638 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:03.638 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:03.780 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:58:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:58:04.865 [conn81] end connection 127.0.0.4:59951 (4 connections now open) m31000| Wed Jan 16 17:58:04.865 [initandlisten] connection accepted from 127.0.0.4:49111 #82 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2128, "optime" : { "t" : 1358377084000, "i" : 8681 }, "optimeDate" : ISODate("2013-01-16T22:58:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2117, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2117, "optime" : { "t" : 1358377080000, "i" : 7014 }, "optimeDate" : ISODate("2013-01-16T22:58:00Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2127, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2115, "optime" : { "t" : 1358377084000, "i" : 1789 }, "optimeDate" : ISODate("2013-01-16T22:58:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2114, "optime" : { "t" : 1358377084000, "i" : 7724 }, "optimeDate" : ISODate("2013-01-16T22:58:04Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2114, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:04Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2126, "optime" : { "t" : 1358377084000, "i" : 8500 }, "optimeDate" : ISODate("2013-01-16T22:58:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:05.638 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:07.530 [conn129] end connection 127.0.0.2:58069 (3 connections now open) m31000| Wed Jan 16 17:58:07.639 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:08.030 [initandlisten] connection accepted from 127.0.0.2:53620 #131 (4 connections now open) m31002| Wed Jan 16 17:58:08.781 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2133, "optime" : { "t" : 1358377089000, "i" : 1190 }, "optimeDate" : ISODate("2013-01-16T22:58:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2122, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2122, "optime" : { "t" : 1358377088000, "i" : 87 }, "optimeDate" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2132, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2120, "optime" : { "t" : 1358377088000, "i" : 1297 }, "optimeDate" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2119, "optime" : { "t" : 1358377088000, "i" : 7611 }, "optimeDate" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2119, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2131, "optime" : { "t" : 1358377089000, "i" : 1190 }, "optimeDate" : ISODate("2013-01-16T22:58:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:09.639 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:58:11.281 [conn128] end connection 127.0.0.4:45544 (2 connections now open) m31000| Wed Jan 16 17:58:11.640 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:58:11.781 [initandlisten] connection accepted from 127.0.0.4:37634 #130 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2137, "optime" : { "t" : 1358377093000, "i" : 3231 }, "optimeDate" : ISODate("2013-01-16T22:58:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2126, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2126, "optime" : { "t" : 1358377093000, "i" : 7 }, "optimeDate" : ISODate("2013-01-16T22:58:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2136, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2124, "optime" : { "t" : 1358377092000, "i" : 219 }, "optimeDate" : ISODate("2013-01-16T22:58:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2123, "optime" : { "t" : 1358377092000, "i" : 5577 }, "optimeDate" : ISODate("2013-01-16T22:58:12Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2123, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2135, "optime" : { "t" : 1358377093000, "i" : 3231 }, "optimeDate" : ISODate("2013-01-16T22:58:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:13.640 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:14.282 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:58:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:58:15.641 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2141, "optime" : { "t" : 1358377097000, "i" : 4636 }, "optimeDate" : ISODate("2013-01-16T22:58:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2130, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2130, "optime" : { "t" : 1358377093000, "i" : 3231 }, "optimeDate" : ISODate("2013-01-16T22:58:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2140, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2128, "optime" : { "t" : 1358377093000, "i" : 3231 }, "optimeDate" : ISODate("2013-01-16T22:58:13Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2127, "optime" : { "t" : 1358377096000, "i" : 4222 }, "optimeDate" : ISODate("2013-01-16T22:58:16Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2127, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2139, "optime" : { "t" : 1358377097000, "i" : 4615 }, "optimeDate" : ISODate("2013-01-16T22:58:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:17.641 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:18.233 [conn130] end connection 127.0.0.3:42725 (3 connections now open) m31002| Wed Jan 16 17:58:18.233 [initandlisten] connection accepted from 127.0.0.3:39098 #132 (4 connections now open) m31002| Wed Jan 16 17:58:19.283 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:58:19.641 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2145, "optime" : { "t" : 1358377101000, "i" : 5273 }, "optimeDate" : ISODate("2013-01-16T22:58:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2134, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2134, "optime" : { "t" : 1358377100000, "i" : 304 }, "optimeDate" : ISODate("2013-01-16T22:58:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2144, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2132, "optime" : { "t" : 1358377097000, "i" : 4636 }, "optimeDate" : ISODate("2013-01-16T22:58:17Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2131, "optime" : { "t" : 1358377100000, "i" : 3457 }, "optimeDate" : ISODate("2013-01-16T22:58:20Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2131, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2143, "optime" : { "t" : 1358377101000, "i" : 4913 }, "optimeDate" : ISODate("2013-01-16T22:58:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:21.642 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:23.642 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:24.283 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:58:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:58:25.642 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2149, "optime" : { "t" : 1358377105000, "i" : 6956 }, "optimeDate" : ISODate("2013-01-16T22:58:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2138, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2138, "optime" : { "t" : 1358377105000, "i" : 4783 }, "optimeDate" : ISODate("2013-01-16T22:58:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2148, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2136, "optime" : { "t" : 1358377101000, "i" : 5273 }, "optimeDate" : ISODate("2013-01-16T22:58:21Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:24Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2135, "optime" : { "t" : 1358377104000, "i" : 1869 }, "optimeDate" : ISODate("2013-01-16T22:58:24Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2135, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2147, "optime" : { "t" : 1358377105000, "i" : 6956 }, "optimeDate" : ISODate("2013-01-16T22:58:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:27.643 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:29.284 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:58:29.477 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:277194 138ms m31000| Wed Jan 16 17:58:29.643 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2153, "optime" : { "t" : 1358377109000, "i" : 8781 }, "optimeDate" : ISODate("2013-01-16T22:58:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2142, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2142, "optime" : { "t" : 1358377105000, "i" : 6956 }, "optimeDate" : ISODate("2013-01-16T22:58:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2152, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2140, "optime" : { "t" : 1358377105000, "i" : 6956 }, "optimeDate" : ISODate("2013-01-16T22:58:25Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:29Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2139, "optime" : { "t" : 1358377108000, "i" : 58 }, "optimeDate" : ISODate("2013-01-16T22:58:28Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2139, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2151, "optime" : { "t" : 1358377109000, "i" : 7970 }, "optimeDate" : ISODate("2013-01-16T22:58:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:58:31.643 [conn129] end connection 127.0.0.2:45980 (2 connections now open) m31001| Wed Jan 16 17:58:31.644 [initandlisten] connection accepted from 127.0.0.2:47539 #131 (3 connections now open) m31000| Wed Jan 16 17:58:31.644 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:33.644 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2158, "optime" : { "t" : 1358377114000, "i" : 754 }, "optimeDate" : ISODate("2013-01-16T22:58:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2147, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2147, "optime" : { "t" : 1358377113000, "i" : 82 }, "optimeDate" : ISODate("2013-01-16T22:58:33Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2157, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2145, "optime" : { "t" : 1358377109000, "i" : 8781 }, "optimeDate" : ISODate("2013-01-16T22:58:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2144, "optime" : { "t" : 1358377109000, "i" : 8781 }, "optimeDate" : ISODate("2013-01-16T22:58:29Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2144, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2156, "optime" : { "t" : 1358377114000, "i" : 32 }, "optimeDate" : ISODate("2013-01-16T22:58:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:58:34.285 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:58:34.870 [conn82] end connection 127.0.0.4:49111 (4 connections now open) m31000| Wed Jan 16 17:58:34.870 [initandlisten] connection accepted from 127.0.0.4:41813 #83 (5 connections now open) m31000| Wed Jan 16 17:58:35.645 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:37.645 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2162, "optime" : { "t" : 1358377118000, "i" : 1834 }, "optimeDate" : ISODate("2013-01-16T22:58:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2151, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2151, "optime" : { "t" : 1358377118000, "i" : 584 }, "optimeDate" : ISODate("2013-01-16T22:58:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2161, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2149, "optime" : { "t" : 1358377114000, "i" : 754 }, "optimeDate" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2148, "optime" : { "t" : 1358377114000, "i" : 754 }, "optimeDate" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2148, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2160, "optime" : { "t" : 1358377118000, "i" : 1647 }, "optimeDate" : ISODate("2013-01-16T22:58:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:58:39.285 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:58:39.645 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:41.645 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2166, "optime" : { "t" : 1358377122000, "i" : 3152 }, "optimeDate" : ISODate("2013-01-16T22:58:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2155, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2155, "optime" : { "t" : 1358377118000, "i" : 1834 }, "optimeDate" : ISODate("2013-01-16T22:58:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2165, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2153, "optime" : { "t" : 1358377122000, "i" : 1722 }, "optimeDate" : ISODate("2013-01-16T22:58:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2152, "optime" : { "t" : 1358377118000, "i" : 1834 }, "optimeDate" : ISODate("2013-01-16T22:58:38Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2152, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2164, "optime" : { "t" : 1358377122000, "i" : 3152 }, "optimeDate" : ISODate("2013-01-16T22:58:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:43.646 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:44.286 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 17:58:45.535 [conn131] end connection 127.0.0.2:53620 (3 connections now open) m31000| Wed Jan 16 17:58:45.646 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:46.035 [initandlisten] connection accepted from 127.0.0.2:43621 #133 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2170, "optime" : { "t" : 1358377126000, "i" : 4930 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2159, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2159, "optime" : { "t" : 1358377126000, "i" : 268 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2169, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2157, "optime" : { "t" : 1358377126000, "i" : 1798 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2156, "optime" : { "t" : 1358377122000, "i" : 3152 }, "optimeDate" : ISODate("2013-01-16T22:58:42Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2156, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2168, "optime" : { "t" : 1358377126000, "i" : 4129 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:58:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:58:47.647 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:48.239 [conn132] end connection 127.0.0.3:39098 (3 connections now open) m31002| Wed Jan 16 17:58:48.239 [initandlisten] connection accepted from 127.0.0.3:55384 #134 (4 connections now open) m31001| Wed Jan 16 17:58:49.286 [conn130] end connection 127.0.0.4:37634 (2 connections now open) m31000| Wed Jan 16 17:58:49.647 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:58:49.787 [initandlisten] connection accepted from 127.0.0.4:38618 #132 (3 connections now open) m31002| Wed Jan 16 17:58:49.787 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2174, "optime" : { "t" : 1358377130000, "i" : 6285 }, "optimeDate" : ISODate("2013-01-16T22:58:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2163, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2163, "optime" : { "t" : 1358377126000, "i" : 4930 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2173, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2161, "optime" : { "t" : 1358377130000, "i" : 1982 }, "optimeDate" : ISODate("2013-01-16T22:58:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2160, "optime" : { "t" : 1358377126000, "i" : 4930 }, "optimeDate" : ISODate("2013-01-16T22:58:46Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2160, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2172, "optime" : { "t" : 1358377130000, "i" : 6285 }, "optimeDate" : ISODate("2013-01-16T22:58:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:51.647 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:58:53.648 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 17:58:54.787 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2178, "optime" : { "t" : 1358377134000, "i" : 8275 }, "optimeDate" : ISODate("2013-01-16T22:58:54Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2167, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2167, "optime" : { "t" : 1358377130000, "i" : 6285 }, "optimeDate" : ISODate("2013-01-16T22:58:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:54Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2177, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2165, "optime" : { "t" : 1358377134000, "i" : 2539 }, "optimeDate" : ISODate("2013-01-16T22:58:54Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2164, "optime" : { "t" : 1358377130000, "i" : 6285 }, "optimeDate" : ISODate("2013-01-16T22:58:50Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2164, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2176, "optime" : { "t" : 1358377134000, "i" : 6860 }, "optimeDate" : ISODate("2013-01-16T22:58:54Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:55.648 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:58:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:58:57.648 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2182, "optime" : { "t" : 1358377138000, "i" : 8489 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2171, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2171, "optime" : { "t" : 1358377138000, "i" : 4492 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2181, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2169, "optime" : { "t" : 1358377138000, "i" : 2024 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:58:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2168, "optime" : { "t" : 1358377138000, "i" : 7597 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2168, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:58:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2180, "optime" : { "t" : 1358377138000, "i" : 8489 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:58:59.649 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:58:59.788 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:59:01.649 [conn131] end connection 127.0.0.2:47539 (2 connections now open) m31001| Wed Jan 16 17:59:01.649 [initandlisten] connection accepted from 127.0.0.2:51361 #133 (3 connections now open) m31000| Wed Jan 16 17:59:01.650 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2187, "optime" : { "t" : 1358377143000, "i" : 1037 }, "optimeDate" : ISODate("2013-01-16T22:59:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2176, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2176, "optime" : { "t" : 1358377138000, "i" : 8489 }, "optimeDate" : ISODate("2013-01-16T22:58:58Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:58:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2186, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2174, "optime" : { "t" : 1358377142000, "i" : 1782 }, "optimeDate" : ISODate("2013-01-16T22:59:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2173, "optime" : { "t" : 1358377142000, "i" : 7288 }, "optimeDate" : ISODate("2013-01-16T22:59:02Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2173, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2185, "optime" : { "t" : 1358377143000, "i" : 1037 }, "optimeDate" : ISODate("2013-01-16T22:59:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:03.650 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:04.789 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:04.876 [conn83] end connection 127.0.0.4:41813 (4 connections now open) m31000| Wed Jan 16 17:59:04.876 [initandlisten] connection accepted from 127.0.0.4:54904 #84 (5 connections now open) m31000| Wed Jan 16 17:59:05.650 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2191, "optime" : { "t" : 1358377147000, "i" : 2077 }, "optimeDate" : ISODate("2013-01-16T22:59:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2180, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2180, "optime" : { "t" : 1358377143000, "i" : 1037 }, "optimeDate" : ISODate("2013-01-16T22:59:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2190, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2178, "optime" : { "t" : 1358377146000, "i" : 892 }, "optimeDate" : ISODate("2013-01-16T22:59:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2177, "optime" : { "t" : 1358377146000, "i" : 6744 }, "optimeDate" : ISODate("2013-01-16T22:59:06Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2177, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2189, "optime" : { "t" : 1358377147000, "i" : 2077 }, "optimeDate" : ISODate("2013-01-16T22:59:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:07.651 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:09.651 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:09.789 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:10.676 [journal] old journal file will be removed: /data/db/testReplSet-0/journal/j._2 m31000| Wed Jan 16 17:59:10.749 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:313459 156ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2195, "optime" : { "t" : 1358377151000, "i" : 3027 }, "optimeDate" : ISODate("2013-01-16T22:59:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2184, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2184, "optime" : { "t" : 1358377151000, "i" : 572 }, "optimeDate" : ISODate("2013-01-16T22:59:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2194, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2182, "optime" : { "t" : 1358377147000, "i" : 2077 }, "optimeDate" : ISODate("2013-01-16T22:59:07Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2181, "optime" : { "t" : 1358377150000, "i" : 5063 }, "optimeDate" : ISODate("2013-01-16T22:59:10Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2181, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2193, "optime" : { "t" : 1358377151000, "i" : 2703 }, "optimeDate" : ISODate("2013-01-16T22:59:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:11.652 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:13.652 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:14.790 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2199, "optime" : { "t" : 1358377155000, "i" : 4404 }, "optimeDate" : ISODate("2013-01-16T22:59:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2188, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2188, "optime" : { "t" : 1358377151000, "i" : 3027 }, "optimeDate" : ISODate("2013-01-16T22:59:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2198, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2186, "optime" : { "t" : 1358377151000, "i" : 3027 }, "optimeDate" : ISODate("2013-01-16T22:59:11Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2185, "optime" : { "t" : 1358377154000, "i" : 4261 }, "optimeDate" : ISODate("2013-01-16T22:59:14Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2185, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2197, "optime" : { "t" : 1358377155000, "i" : 4404 }, "optimeDate" : ISODate("2013-01-16T22:59:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:15.652 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:17.653 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:18.244 [conn134] end connection 127.0.0.3:55384 (3 connections now open) m31002| Wed Jan 16 17:59:18.244 [initandlisten] connection accepted from 127.0.0.3:58122 #135 (4 connections now open) m31001| Wed Jan 16 17:59:18.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:59:19.653 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2203, "optime" : { "t" : 1358377159000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:59:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2192, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2192, "optime" : { "t" : 1358377155000, "i" : 4404 }, "optimeDate" : ISODate("2013-01-16T22:59:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2202, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2190, "optime" : { "t" : 1358377155000, "i" : 4404 }, "optimeDate" : ISODate("2013-01-16T22:59:15Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2189, "optime" : { "t" : 1358377158000, "i" : 2932 }, "optimeDate" : ISODate("2013-01-16T22:59:18Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2189, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2201, "optime" : { "t" : 1358377159000, "i" : 5617 }, "optimeDate" : ISODate("2013-01-16T22:59:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:59:19.791 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:21.653 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:23.541 [conn133] end connection 127.0.0.2:43621 (3 connections now open) m31000| Wed Jan 16 17:59:23.654 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2207, "optime" : { "t" : 1358377163000, "i" : 7278 }, "optimeDate" : ISODate("2013-01-16T22:59:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2196, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2196, "optime" : { "t" : 1358377159000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:59:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2206, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2194, "optime" : { "t" : 1358377159000, "i" : 5898 }, "optimeDate" : ISODate("2013-01-16T22:59:19Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2193, "optime" : { "t" : 1358377162000, "i" : 1601 }, "optimeDate" : ISODate("2013-01-16T22:59:22Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2193, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2205, "optime" : { "t" : 1358377163000, "i" : 7197 }, "optimeDate" : ISODate("2013-01-16T22:59:23Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:59:24.041 [initandlisten] connection accepted from 127.0.0.2:52266 #136 (4 connections now open) m31002| Wed Jan 16 17:59:24.791 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:25.654 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:59:27.292 [conn132] end connection 127.0.0.4:38618 (2 connections now open) m31000| Wed Jan 16 17:59:27.654 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 17:59:27.792 [initandlisten] connection accepted from 127.0.0.4:38169 #134 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2211, "optime" : { "t" : 1358377167000, "i" : 8988 }, "optimeDate" : ISODate("2013-01-16T22:59:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2200, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2200, "optime" : { "t" : 1358377163000, "i" : 7278 }, "optimeDate" : ISODate("2013-01-16T22:59:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2210, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2198, "optime" : { "t" : 1358377163000, "i" : 7278 }, "optimeDate" : ISODate("2013-01-16T22:59:23Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:27Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2197, "optime" : { "t" : 1358377166000, "i" : 232 }, "optimeDate" : ISODate("2013-01-16T22:59:26Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2197, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2209, "optime" : { "t" : 1358377167000, "i" : 8988 }, "optimeDate" : ISODate("2013-01-16T22:59:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 17:59:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:59:29.655 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:30.292 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:59:31.655 [conn133] end connection 127.0.0.2:51361 (2 connections now open) m31001| Wed Jan 16 17:59:31.655 [initandlisten] connection accepted from 127.0.0.2:58358 #135 (3 connections now open) m31000| Wed Jan 16 17:59:31.655 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2216, "optime" : { "t" : 1358377172000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:59:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2205, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2205, "optime" : { "t" : 1358377171000, "i" : 4658 }, "optimeDate" : ISODate("2013-01-16T22:59:31Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2215, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2203, "optime" : { "t" : 1358377167000, "i" : 8988 }, "optimeDate" : ISODate("2013-01-16T22:59:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2202, "optime" : { "t" : 1358377167000, "i" : 8988 }, "optimeDate" : ISODate("2013-01-16T22:59:27Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2202, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2214, "optime" : { "t" : 1358377172000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:59:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:33.656 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:34.881 [conn84] end connection 127.0.0.4:54904 (4 connections now open) m31000| Wed Jan 16 17:59:34.881 [initandlisten] connection accepted from 127.0.0.4:43601 #85 (5 connections now open) m31002| Wed Jan 16 17:59:35.293 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:35.656 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2220, "optime" : { "t" : 1358377176000, "i" : 2703 }, "optimeDate" : ISODate("2013-01-16T22:59:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2209, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2209, "optime" : { "t" : 1358377172000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:59:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2219, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2207, "optime" : { "t" : 1358377176000, "i" : 2074 }, "optimeDate" : ISODate("2013-01-16T22:59:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2206, "optime" : { "t" : 1358377172000, "i" : 1411 }, "optimeDate" : ISODate("2013-01-16T22:59:32Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2206, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2218, "optime" : { "t" : 1358377176000, "i" : 2703 }, "optimeDate" : ISODate("2013-01-16T22:59:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:37.656 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:39.657 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:40.294 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2224, "optime" : { "t" : 1358377180000, "i" : 4326 }, "optimeDate" : ISODate("2013-01-16T22:59:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2213, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2213, "optime" : { "t" : 1358377176000, "i" : 2703 }, "optimeDate" : ISODate("2013-01-16T22:59:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2223, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2211, "optime" : { "t" : 1358377180000, "i" : 1629 }, "optimeDate" : ISODate("2013-01-16T22:59:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2210, "optime" : { "t" : 1358377176000, "i" : 2703 }, "optimeDate" : ISODate("2013-01-16T22:59:36Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2210, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2222, "optime" : { "t" : 1358377180000, "i" : 4326 }, "optimeDate" : ISODate("2013-01-16T22:59:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:41.657 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:43.657 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2228, "optime" : { "t" : 1358377184000, "i" : 5537 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2217, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2217, "optime" : { "t" : 1358377184000, "i" : 171 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2227, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2215, "optime" : { "t" : 1358377184000, "i" : 1756 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2214, "optime" : { "t" : 1358377180000, "i" : 4326 }, "optimeDate" : ISODate("2013-01-16T22:59:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:44Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2214, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2226, "optime" : { "t" : 1358377184000, "i" : 5537 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 17:59:45.294 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:45.658 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:47.658 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:48.249 [conn135] end connection 127.0.0.3:58122 (3 connections now open) m31002| Wed Jan 16 17:59:48.249 [initandlisten] connection accepted from 127.0.0.3:60447 #137 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2232, "optime" : { "t" : 1358377188000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:59:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2221, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2221, "optime" : { "t" : 1358377184000, "i" : 5537 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2231, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2219, "optime" : { "t" : 1358377188000, "i" : 1892 }, "optimeDate" : ISODate("2013-01-16T22:59:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2218, "optime" : { "t" : 1358377184000, "i" : 5537 }, "optimeDate" : ISODate("2013-01-16T22:59:44Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2218, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2230, "optime" : { "t" : 1358377188000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:59:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:49.658 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:50.295 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 17:59:50.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 17:59:51.659 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:52.290 [journal] old journal file will be removed: /data/db/testReplSet-2/journal/j._2 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2236, "optime" : { "t" : 1358377192000, "i" : 7444 }, "optimeDate" : ISODate("2013-01-16T22:59:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2225, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2225, "optime" : { "t" : 1358377188000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:59:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2235, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2223, "optime" : { "t" : 1358377192000, "i" : 1687 }, "optimeDate" : ISODate("2013-01-16T22:59:52Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2222, "optime" : { "t" : 1358377188000, "i" : 6465 }, "optimeDate" : ISODate("2013-01-16T22:59:48Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2222, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2234, "optime" : { "t" : 1358377192000, "i" : 7185 }, "optimeDate" : ISODate("2013-01-16T22:59:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:53.659 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 17:59:55.296 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 17:59:55.660 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2240, "optime" : { "t" : 1358377196000, "i" : 8262 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2229, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2229, "optime" : { "t" : 1358377196000, "i" : 4679 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2239, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2227, "optime" : { "t" : 1358377196000, "i" : 2259 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T22:59:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2226, "optime" : { "t" : 1358377196000, "i" : 7538 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2226, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:56Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2238, "optime" : { "t" : 1358377196000, "i" : 8262 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 17:59:57.660 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 17:59:59.660 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:00.296 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:00:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2245, "optime" : { "t" : 1358377201000, "i" : 1099 }, "optimeDate" : ISODate("2013-01-16T23:00:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2234, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2234, "optime" : { "t" : 1358377196000, "i" : 8262 }, "optimeDate" : ISODate("2013-01-16T22:59:56Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T22:59:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2244, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2232, "optime" : { "t" : 1358377200000, "i" : 1964 }, "optimeDate" : ISODate("2013-01-16T23:00:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2231, "optime" : { "t" : 1358377200000, "i" : 7674 }, "optimeDate" : ISODate("2013-01-16T23:00:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2231, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2243, "optime" : { "t" : 1358377201000, "i" : 1099 }, "optimeDate" : ISODate("2013-01-16T23:00:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:00:01.546 [conn136] end connection 127.0.0.2:52266 (3 connections now open) m31001| Wed Jan 16 18:00:01.661 [conn135] end connection 127.0.0.2:58358 (2 connections now open) m31001| Wed Jan 16 18:00:01.661 [initandlisten] connection accepted from 127.0.0.2:53120 #136 (3 connections now open) m31000| Wed Jan 16 18:00:01.661 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:02.046 [initandlisten] connection accepted from 127.0.0.2:53328 #138 (4 connections now open) m31000| Wed Jan 16 18:00:03.661 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:04.657 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:281424 281ms m31000| Wed Jan 16 18:00:04.886 [conn85] end connection 127.0.0.4:43601 (4 connections now open) m31000| Wed Jan 16 18:00:04.886 [initandlisten] connection accepted from 127.0.0.4:59071 #86 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2249, "optime" : { "t" : 1358377205000, "i" : 1652 }, "optimeDate" : ISODate("2013-01-16T23:00:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2238, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2238, "optime" : { "t" : 1358377204000, "i" : 2162 }, "optimeDate" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2248, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2236, "optime" : { "t" : 1358377204000, "i" : 809 }, "optimeDate" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2235, "optime" : { "t" : 1358377204000, "i" : 6889 }, "optimeDate" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2235, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:04Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2247, "optime" : { "t" : 1358377205000, "i" : 1652 }, "optimeDate" : ISODate("2013-01-16T23:00:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:00:05.297 [conn134] end connection 127.0.0.4:38169 (2 connections now open) m31000| Wed Jan 16 18:00:05.662 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:00:05.797 [initandlisten] connection accepted from 127.0.0.4:44909 #137 (3 connections now open) m31002| Wed Jan 16 18:00:05.797 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:07.662 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2253, "optime" : { "t" : 1358377209000, "i" : 3062 }, "optimeDate" : ISODate("2013-01-16T23:00:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2242, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2242, "optime" : { "t" : 1358377209000, "i" : 3023 }, "optimeDate" : ISODate("2013-01-16T23:00:09Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T22:59:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2252, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2240, "optime" : { "t" : 1358377208000, "i" : 95 }, "optimeDate" : ISODate("2013-01-16T23:00:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2239, "optime" : { "t" : 1358377208000, "i" : 5852 }, "optimeDate" : ISODate("2013-01-16T23:00:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2239, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2251, "optime" : { "t" : 1358377209000, "i" : 3023 }, "optimeDate" : ISODate("2013-01-16T23:00:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:09.663 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:10.798 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:11.663 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:13.663 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2257, "optime" : { "t" : 1358377213000, "i" : 6481 }, "optimeDate" : ISODate("2013-01-16T23:00:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2246, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2246, "optime" : { "t" : 1358377209000, "i" : 3062 }, "optimeDate" : ISODate("2013-01-16T23:00:09Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2256, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2244, "optime" : { "t" : 1358377209000, "i" : 3062 }, "optimeDate" : ISODate("2013-01-16T23:00:09Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2243, "optime" : { "t" : 1358377212000, "i" : 2112 }, "optimeDate" : ISODate("2013-01-16T23:00:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2243, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2255, "optime" : { "t" : 1358377213000, "i" : 6371 }, "optimeDate" : ISODate("2013-01-16T23:00:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:15.664 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:15.798 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:17.664 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2261, "optime" : { "t" : 1358377217000, "i" : 7436 }, "optimeDate" : ISODate("2013-01-16T23:00:17Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2250, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2250, "optime" : { "t" : 1358377217000, "i" : 307 }, "optimeDate" : ISODate("2013-01-16T23:00:17Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:17Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2260, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2248, "optime" : { "t" : 1358377213000, "i" : 6481 }, "optimeDate" : ISODate("2013-01-16T23:00:13Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:17Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2247, "optime" : { "t" : 1358377216000, "i" : 1423 }, "optimeDate" : ISODate("2013-01-16T23:00:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2247, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2259, "optime" : { "t" : 1358377217000, "i" : 7436 }, "optimeDate" : ISODate("2013-01-16T23:00:17Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:00:18.254 [conn137] end connection 127.0.0.3:60447 (3 connections now open) m31002| Wed Jan 16 18:00:18.254 [initandlisten] connection accepted from 127.0.0.3:52382 #139 (4 connections now open) m31000| Wed Jan 16 18:00:19.664 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:20.799 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:21.665 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2265, "optime" : { "t" : 1358377221000, "i" : 8359 }, "optimeDate" : ISODate("2013-01-16T23:00:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2254, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2254, "optime" : { "t" : 1358377217000, "i" : 7436 }, "optimeDate" : ISODate("2013-01-16T23:00:17Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:20Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2264, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2252, "optime" : { "t" : 1358377217000, "i" : 7436 }, "optimeDate" : ISODate("2013-01-16T23:00:17Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2251, "optime" : { "t" : 1358377220000, "i" : 99 }, "optimeDate" : ISODate("2013-01-16T23:00:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2251, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2263, "optime" : { "t" : 1358377221000, "i" : 8025 }, "optimeDate" : ISODate("2013-01-16T23:00:21Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:00:22.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:00:23.665 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:25.665 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:25.799 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2270, "optime" : { "t" : 1358377226000, "i" : 420 }, "optimeDate" : ISODate("2013-01-16T23:00:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2259, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2259, "optime" : { "t" : 1358377221000, "i" : 8359 }, "optimeDate" : ISODate("2013-01-16T23:00:21Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2269, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2257, "optime" : { "t" : 1358377221000, "i" : 8359 }, "optimeDate" : ISODate("2013-01-16T23:00:21Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2256, "optime" : { "t" : 1358377224000, "i" : 8 }, "optimeDate" : ISODate("2013-01-16T23:00:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2256, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2268, "optime" : { "t" : 1358377226000, "i" : 420 }, "optimeDate" : ISODate("2013-01-16T23:00:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:27.666 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:29.666 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2274, "optime" : { "t" : 1358377230000, "i" : 1792 }, "optimeDate" : ISODate("2013-01-16T23:00:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2263, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2263, "optime" : { "t" : 1358377229000, "i" : 3796 }, "optimeDate" : ISODate("2013-01-16T23:00:29Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2273, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2261, "optime" : { "t" : 1358377226000, "i" : 420 }, "optimeDate" : ISODate("2013-01-16T23:00:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2260, "optime" : { "t" : 1358377226000, "i" : 420 }, "optimeDate" : ISODate("2013-01-16T23:00:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2260, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2272, "optime" : { "t" : 1358377230000, "i" : 1648 }, "optimeDate" : ISODate("2013-01-16T23:00:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:00:30.800 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:00:31.666 [conn136] end connection 127.0.0.2:53120 (2 connections now open) m31001| Wed Jan 16 18:00:31.667 [initandlisten] connection accepted from 127.0.0.2:51015 #138 (3 connections now open) m31000| Wed Jan 16 18:00:31.667 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:00:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:00:33.667 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2278, "optime" : { "t" : 1358377234000, "i" : 2823 }, "optimeDate" : ISODate("2013-01-16T23:00:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2267, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2267, "optime" : { "t" : 1358377230000, "i" : 1792 }, "optimeDate" : ISODate("2013-01-16T23:00:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2277, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2265, "optime" : { "t" : 1358377234000, "i" : 2229 }, "optimeDate" : ISODate("2013-01-16T23:00:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2264, "optime" : { "t" : 1358377230000, "i" : 1792 }, "optimeDate" : ISODate("2013-01-16T23:00:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2264, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2276, "optime" : { "t" : 1358377234000, "i" : 2823 }, "optimeDate" : ISODate("2013-01-16T23:00:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:34.892 [conn86] end connection 127.0.0.4:59071 (4 connections now open) m31000| Wed Jan 16 18:00:34.892 [initandlisten] connection accepted from 127.0.0.4:58361 #87 (5 connections now open) m31000| Wed Jan 16 18:00:35.667 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:35.801 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:37.668 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:38.247 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:276957 138ms STATUS FROM : 127.0.0.2:31000 m31000| Wed Jan 16 18:00:38.455 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:277989 139ms { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2282, "optime" : { "t" : 1358377238000, "i" : 4828 }, "optimeDate" : ISODate("2013-01-16T23:00:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2271, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2271, "optime" : { "t" : 1358377234000, "i" : 2823 }, "optimeDate" : ISODate("2013-01-16T23:00:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2281, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2269, "optime" : { "t" : 1358377238000, "i" : 1405 }, "optimeDate" : ISODate("2013-01-16T23:00:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2268, "optime" : { "t" : 1358377234000, "i" : 2823 }, "optimeDate" : ISODate("2013-01-16T23:00:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2268, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2280, "optime" : { "t" : 1358377238000, "i" : 4828 }, "optimeDate" : ISODate("2013-01-16T23:00:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:00:39.552 [conn138] end connection 127.0.0.2:53328 (3 connections now open) m31000| Wed Jan 16 18:00:39.668 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:40.052 [initandlisten] connection accepted from 127.0.0.2:44725 #140 (4 connections now open) m31002| Wed Jan 16 18:00:40.802 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:41.668 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2286, "optime" : { "t" : 1358377242000, "i" : 5889 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2275, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2275, "optime" : { "t" : 1358377242000, "i" : 4785 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2285, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2273, "optime" : { "t" : 1358377242000, "i" : 1747 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2272, "optime" : { "t" : 1358377238000, "i" : 4828 }, "optimeDate" : ISODate("2013-01-16T23:00:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2272, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2284, "optime" : { "t" : 1358377242000, "i" : 5889 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:00:43.302 [conn137] end connection 127.0.0.4:44909 (2 connections now open) m31000| Wed Jan 16 18:00:43.669 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:00:43.802 [initandlisten] connection accepted from 127.0.0.4:57782 #139 (3 connections now open) m31000| Wed Jan 16 18:00:45.669 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:46.303 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2290, "optime" : { "t" : 1358377246000, "i" : 7435 }, "optimeDate" : ISODate("2013-01-16T23:00:46Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2279, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2279, "optime" : { "t" : 1358377242000, "i" : 5889 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:46Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2289, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2277, "optime" : { "t" : 1358377246000, "i" : 2536 }, "optimeDate" : ISODate("2013-01-16T23:00:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:46Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2276, "optime" : { "t" : 1358377242000, "i" : 5889 }, "optimeDate" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2276, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2288, "optime" : { "t" : 1358377246000, "i" : 7435 }, "optimeDate" : ISODate("2013-01-16T23:00:46Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:47.669 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:48.260 [conn139] end connection 127.0.0.3:52382 (3 connections now open) m31002| Wed Jan 16 18:00:48.260 [initandlisten] connection accepted from 127.0.0.3:41340 #141 (4 connections now open) m31000| Wed Jan 16 18:00:49.670 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2294, "optime" : { "t" : 1358377250000, "i" : 8785 }, "optimeDate" : ISODate("2013-01-16T23:00:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2283, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2283, "optime" : { "t" : 1358377250000, "i" : 233 }, "optimeDate" : ISODate("2013-01-16T23:00:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2293, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2281, "optime" : { "t" : 1358377250000, "i" : 1549 }, "optimeDate" : ISODate("2013-01-16T23:00:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2280, "optime" : { "t" : 1358377250000, "i" : 7479 }, "optimeDate" : ISODate("2013-01-16T23:00:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2280, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2292, "optime" : { "t" : 1358377250000, "i" : 8785 }, "optimeDate" : ISODate("2013-01-16T23:00:50Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:00:51.303 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:51.670 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:00:53.671 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:00:54.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2299, "optime" : { "t" : 1358377255000, "i" : 1428 }, "optimeDate" : ISODate("2013-01-16T23:00:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2288, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2288, "optime" : { "t" : 1358377255000, "i" : 281 }, "optimeDate" : ISODate("2013-01-16T23:00:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2298, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2286, "optime" : { "t" : 1358377254000, "i" : 2014 }, "optimeDate" : ISODate("2013-01-16T23:00:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2285, "optime" : { "t" : 1358377254000, "i" : 7544 }, "optimeDate" : ISODate("2013-01-16T23:00:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2285, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2297, "optime" : { "t" : 1358377255000, "i" : 1428 }, "optimeDate" : ISODate("2013-01-16T23:00:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:55.671 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:00:56.304 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:00:57.672 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2303, "optime" : { "t" : 1358377259000, "i" : 2546 }, "optimeDate" : ISODate("2013-01-16T23:00:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2292, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2292, "optime" : { "t" : 1358377255000, "i" : 1428 }, "optimeDate" : ISODate("2013-01-16T23:00:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2302, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2290, "optime" : { "t" : 1358377258000, "i" : 335 }, "optimeDate" : ISODate("2013-01-16T23:00:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:00:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:00:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2289, "optime" : { "t" : 1358377258000, "i" : 6403 }, "optimeDate" : ISODate("2013-01-16T23:00:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2289, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2301, "optime" : { "t" : 1358377259000, "i" : 2546 }, "optimeDate" : ISODate("2013-01-16T23:00:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:00:59.672 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:01.304 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:01:01.672 [conn138] end connection 127.0.0.2:51015 (2 connections now open) m31001| Wed Jan 16 18:01:01.672 [initandlisten] connection accepted from 127.0.0.2:55917 #140 (3 connections now open) m31000| Wed Jan 16 18:01:01.673 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2307, "optime" : { "t" : 1358377263000, "i" : 3626 }, "optimeDate" : ISODate("2013-01-16T23:01:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2296, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2296, "optime" : { "t" : 1358377262000, "i" : 2002 }, "optimeDate" : ISODate("2013-01-16T23:01:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2306, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2294, "optime" : { "t" : 1358377259000, "i" : 2546 }, "optimeDate" : ISODate("2013-01-16T23:00:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2293, "optime" : { "t" : 1358377262000, "i" : 5313 }, "optimeDate" : ISODate("2013-01-16T23:01:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2293, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2305, "optime" : { "t" : 1358377263000, "i" : 3454 }, "optimeDate" : ISODate("2013-01-16T23:01:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:03.673 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:01:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:01:04.897 [conn87] end connection 127.0.0.4:58361 (4 connections now open) m31000| Wed Jan 16 18:01:04.897 [initandlisten] connection accepted from 127.0.0.4:53105 #88 (5 connections now open) m31000| Wed Jan 16 18:01:05.673 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:06.305 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2311, "optime" : { "t" : 1358377267000, "i" : 5118 }, "optimeDate" : ISODate("2013-01-16T23:01:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2300, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2300, "optime" : { "t" : 1358377267000, "i" : 4900 }, "optimeDate" : ISODate("2013-01-16T23:01:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2310, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2298, "optime" : { "t" : 1358377263000, "i" : 3626 }, "optimeDate" : ISODate("2013-01-16T23:01:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2297, "optime" : { "t" : 1358377266000, "i" : 3994 }, "optimeDate" : ISODate("2013-01-16T23:01:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2297, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2309, "optime" : { "t" : 1358377267000, "i" : 5118 }, "optimeDate" : ISODate("2013-01-16T23:01:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:07.674 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:09.674 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:11.306 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:01:11.674 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2315, "optime" : { "t" : 1358377271000, "i" : 6686 }, "optimeDate" : ISODate("2013-01-16T23:01:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2304, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2304, "optime" : { "t" : 1358377267000, "i" : 5118 }, "optimeDate" : ISODate("2013-01-16T23:01:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:00:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2314, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2302, "optime" : { "t" : 1358377267000, "i" : 5118 }, "optimeDate" : ISODate("2013-01-16T23:01:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2301, "optime" : { "t" : 1358377270000, "i" : 2638 }, "optimeDate" : ISODate("2013-01-16T23:01:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2301, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2313, "optime" : { "t" : 1358377271000, "i" : 6179 }, "optimeDate" : ISODate("2013-01-16T23:01:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:13.675 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:15.675 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2319, "optime" : { "t" : 1358377275000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T23:01:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2308, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2308, "optime" : { "t" : 1358377275000, "i" : 888 }, "optimeDate" : ISODate("2013-01-16T23:01:15Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2318, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2306, "optime" : { "t" : 1358377271000, "i" : 6686 }, "optimeDate" : ISODate("2013-01-16T23:01:11Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2305, "optime" : { "t" : 1358377274000, "i" : 1212 }, "optimeDate" : ISODate("2013-01-16T23:01:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2305, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2317, "optime" : { "t" : 1358377275000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T23:01:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:01:16.307 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:01:17.557 [conn140] end connection 127.0.0.2:44725 (3 connections now open) m31000| Wed Jan 16 18:01:17.675 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:18.057 [initandlisten] connection accepted from 127.0.0.2:44145 #142 (4 connections now open) m31002| Wed Jan 16 18:01:18.266 [conn141] end connection 127.0.0.3:41340 (3 connections now open) m31002| Wed Jan 16 18:01:18.266 [initandlisten] connection accepted from 127.0.0.3:45922 #143 (4 connections now open) m31000| Wed Jan 16 18:01:19.676 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2324, "optime" : { "t" : 1358377280000, "i" : 397 }, "optimeDate" : ISODate("2013-01-16T23:01:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2313, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2313, "optime" : { "t" : 1358377275000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T23:01:15Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2323, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2311, "optime" : { "t" : 1358377275000, "i" : 8224 }, "optimeDate" : ISODate("2013-01-16T23:01:15Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2310, "optime" : { "t" : 1358377278000, "i" : 89 }, "optimeDate" : ISODate("2013-01-16T23:01:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2310, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2322, "optime" : { "t" : 1358377279000, "i" : 8958 }, "optimeDate" : ISODate("2013-01-16T23:01:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:01:21.307 [conn139] end connection 127.0.0.4:57782 (2 connections now open) m31000| Wed Jan 16 18:01:21.676 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:01:21.807 [initandlisten] connection accepted from 127.0.0.4:42261 #141 (3 connections now open) m31002| Wed Jan 16 18:01:21.808 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:01:23.676 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2328, "optime" : { "t" : 1358377284000, "i" : 1203 }, "optimeDate" : ISODate("2013-01-16T23:01:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2317, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2317, "optime" : { "t" : 1358377283000, "i" : 109 }, "optimeDate" : ISODate("2013-01-16T23:01:23Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2327, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2315, "optime" : { "t" : 1358377280000, "i" : 397 }, "optimeDate" : ISODate("2013-01-16T23:01:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2314, "optime" : { "t" : 1358377280000, "i" : 397 }, "optimeDate" : ISODate("2013-01-16T23:01:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2314, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2326, "optime" : { "t" : 1358377284000, "i" : 1203 }, "optimeDate" : ISODate("2013-01-16T23:01:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:25.677 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:26.808 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:01:26.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:01:27.677 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2332, "optime" : { "t" : 1358377288000, "i" : 2688 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2321, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2321, "optime" : { "t" : 1358377288000, "i" : 389 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2331, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2319, "optime" : { "t" : 1358377288000, "i" : 2328 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2318, "optime" : { "t" : 1358377284000, "i" : 1203 }, "optimeDate" : ISODate("2013-01-16T23:01:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2318, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2330, "optime" : { "t" : 1358377288000, "i" : 2688 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:29.677 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:01:31.678 [conn140] end connection 127.0.0.2:55917 (2 connections now open) m31001| Wed Jan 16 18:01:31.678 [initandlisten] connection accepted from 127.0.0.2:56570 #142 (3 connections now open) m31000| Wed Jan 16 18:01:31.678 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:31.809 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2336, "optime" : { "t" : 1358377292000, "i" : 4332 }, "optimeDate" : ISODate("2013-01-16T23:01:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2325, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2325, "optime" : { "t" : 1358377288000, "i" : 2688 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2335, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2323, "optime" : { "t" : 1358377292000, "i" : 1752 }, "optimeDate" : ISODate("2013-01-16T23:01:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2322, "optime" : { "t" : 1358377288000, "i" : 2688 }, "optimeDate" : ISODate("2013-01-16T23:01:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2322, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2334, "optime" : { "t" : 1358377292000, "i" : 4332 }, "optimeDate" : ISODate("2013-01-16T23:01:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:33.678 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:34.902 [conn88] end connection 127.0.0.4:53105 (4 connections now open) m31000| Wed Jan 16 18:01:34.903 [initandlisten] connection accepted from 127.0.0.4:57718 #89 (5 connections now open) m31000| Wed Jan 16 18:01:35.679 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2340, "optime" : { "t" : 1358377296000, "i" : 5951 }, "optimeDate" : ISODate("2013-01-16T23:01:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2329, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2329, "optime" : { "t" : 1358377295000, "i" : 36 }, "optimeDate" : ISODate("2013-01-16T23:01:35Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2339, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2327, "optime" : { "t" : 1358377296000, "i" : 1753 }, "optimeDate" : ISODate("2013-01-16T23:01:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2326, "optime" : { "t" : 1358377292000, "i" : 4332 }, "optimeDate" : ISODate("2013-01-16T23:01:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2326, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2338, "optime" : { "t" : 1358377296000, "i" : 5951 }, "optimeDate" : ISODate("2013-01-16T23:01:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:01:36.810 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:01:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:01:37.679 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:39.680 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2344, "optime" : { "t" : 1358377300000, "i" : 7946 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2333, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2333, "optime" : { "t" : 1358377300000, "i" : 4395 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2343, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2331, "optime" : { "t" : 1358377300000, "i" : 1586 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2330, "optime" : { "t" : 1358377300000, "i" : 7915 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2330, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2342, "optime" : { "t" : 1358377300000, "i" : 7946 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:41.680 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:41.810 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:01:43.680 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:44.189 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:415964 207ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2348, "optime" : { "t" : 1358377304000, "i" : 9778 }, "optimeDate" : ISODate("2013-01-16T23:01:44Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2337, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2337, "optime" : { "t" : 1358377300000, "i" : 7946 }, "optimeDate" : ISODate("2013-01-16T23:01:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2347, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2335, "optime" : { "t" : 1358377304000, "i" : 1715 }, "optimeDate" : ISODate("2013-01-16T23:01:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:44Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2334, "optime" : { "t" : 1358377304000, "i" : 8782 }, "optimeDate" : ISODate("2013-01-16T23:01:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2334, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2346, "optime" : { "t" : 1358377304000, "i" : 9778 }, "optimeDate" : ISODate("2013-01-16T23:01:44Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:45.681 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:46.811 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:01:47.681 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:48.271 [conn143] end connection 127.0.0.3:45922 (3 connections now open) m31002| Wed Jan 16 18:01:48.271 [initandlisten] connection accepted from 127.0.0.3:43977 #144 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2353, "optime" : { "t" : 1358377309000, "i" : 863 }, "optimeDate" : ISODate("2013-01-16T23:01:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2342, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2342, "optime" : { "t" : 1358377308000, "i" : 534 }, "optimeDate" : ISODate("2013-01-16T23:01:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2352, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2340, "optime" : { "t" : 1358377308000, "i" : 2399 }, "optimeDate" : ISODate("2013-01-16T23:01:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2339, "optime" : { "t" : 1358377308000, "i" : 8131 }, "optimeDate" : ISODate("2013-01-16T23:01:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2339, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2351, "optime" : { "t" : 1358377309000, "i" : 863 }, "optimeDate" : ISODate("2013-01-16T23:01:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:49.681 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:01:51.682 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:51.812 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2357, "optime" : { "t" : 1358377313000, "i" : 2216 }, "optimeDate" : ISODate("2013-01-16T23:01:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2346, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2346, "optime" : { "t" : 1358377313000, "i" : 372 }, "optimeDate" : ISODate("2013-01-16T23:01:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2356, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2344, "optime" : { "t" : 1358377312000, "i" : 522 }, "optimeDate" : ISODate("2013-01-16T23:01:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2343, "optime" : { "t" : 1358377312000, "i" : 7242 }, "optimeDate" : ISODate("2013-01-16T23:01:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2343, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2355, "optime" : { "t" : 1358377313000, "i" : 2216 }, "optimeDate" : ISODate("2013-01-16T23:01:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:53.682 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:55.562 [conn142] end connection 127.0.0.2:44145 (3 connections now open) m31000| Wed Jan 16 18:01:55.683 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:01:56.062 [initandlisten] connection accepted from 127.0.0.2:52492 #145 (4 connections now open) m31002| Wed Jan 16 18:01:56.812 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2361, "optime" : { "t" : 1358377317000, "i" : 3506 }, "optimeDate" : ISODate("2013-01-16T23:01:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2350, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2350, "optime" : { "t" : 1358377313000, "i" : 2216 }, "optimeDate" : ISODate("2013-01-16T23:01:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2360, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2348, "optime" : { "t" : 1358377316000, "i" : 27 }, "optimeDate" : ISODate("2013-01-16T23:01:56Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:01:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2347, "optime" : { "t" : 1358377316000, "i" : 5725 }, "optimeDate" : ISODate("2013-01-16T23:01:56Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2347, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2359, "optime" : { "t" : 1358377317000, "i" : 3506 }, "optimeDate" : ISODate("2013-01-16T23:01:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:01:57.683 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:01:58.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 18:01:59.312 [conn141] end connection 127.0.0.4:42261 (2 connections now open) m31000| Wed Jan 16 18:01:59.684 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:01:59.812 [initandlisten] connection accepted from 127.0.0.4:33888 #143 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2365, "optime" : { "t" : 1358377321000, "i" : 5160 }, "optimeDate" : ISODate("2013-01-16T23:02:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2354, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2354, "optime" : { "t" : 1358377321000, "i" : 476 }, "optimeDate" : ISODate("2013-01-16T23:02:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:01:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2364, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2352, "optime" : { "t" : 1358377317000, "i" : 3506 }, "optimeDate" : ISODate("2013-01-16T23:01:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2351, "optime" : { "t" : 1358377320000, "i" : 3955 }, "optimeDate" : ISODate("2013-01-16T23:02:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2351, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2363, "optime" : { "t" : 1358377321000, "i" : 4906 }, "optimeDate" : ISODate("2013-01-16T23:02:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:02:01.684 [conn142] end connection 127.0.0.2:56570 (2 connections now open) m31001| Wed Jan 16 18:02:01.684 [initandlisten] connection accepted from 127.0.0.2:60981 #144 (3 connections now open) m31000| Wed Jan 16 18:02:01.684 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:02.313 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:03.684 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:04.907 [conn89] end connection 127.0.0.4:57718 (4 connections now open) m31000| Wed Jan 16 18:02:04.907 [initandlisten] connection accepted from 127.0.0.4:55563 #90 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2369, "optime" : { "t" : 1358377325000, "i" : 5808 }, "optimeDate" : ISODate("2013-01-16T23:02:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2358, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2358, "optime" : { "t" : 1358377321000, "i" : 5160 }, "optimeDate" : ISODate("2013-01-16T23:02:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2368, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2356, "optime" : { "t" : 1358377321000, "i" : 5160 }, "optimeDate" : ISODate("2013-01-16T23:02:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2355, "optime" : { "t" : 1358377324000, "i" : 3184 }, "optimeDate" : ISODate("2013-01-16T23:02:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2355, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2367, "optime" : { "t" : 1358377325000, "i" : 5808 }, "optimeDate" : ISODate("2013-01-16T23:02:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:05.685 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:07.314 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:07.685 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:02:08.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:02:09.685 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2373, "optime" : { "t" : 1358377329000, "i" : 6939 }, "optimeDate" : ISODate("2013-01-16T23:02:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2362, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2362, "optime" : { "t" : 1358377325000, "i" : 5808 }, "optimeDate" : ISODate("2013-01-16T23:02:05Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2372, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2360, "optime" : { "t" : 1358377325000, "i" : 5808 }, "optimeDate" : ISODate("2013-01-16T23:02:05Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2359, "optime" : { "t" : 1358377328000, "i" : 2035 }, "optimeDate" : ISODate("2013-01-16T23:02:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2359, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2371, "optime" : { "t" : 1358377329000, "i" : 6939 }, "optimeDate" : ISODate("2013-01-16T23:02:09Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:11.686 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:12.314 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:13.686 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2377, "optime" : { "t" : 1358377333000, "i" : 8505 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2366, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2366, "optime" : { "t" : 1358377333000, "i" : 4478 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2376, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2364, "optime" : { "t" : 1358377329000, "i" : 6939 }, "optimeDate" : ISODate("2013-01-16T23:02:09Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:12Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2363, "optime" : { "t" : 1358377332000, "i" : 250 }, "optimeDate" : ISODate("2013-01-16T23:02:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2363, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2375, "optime" : { "t" : 1358377333000, "i" : 8452 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:15.687 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:17.315 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:17.687 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2382, "optime" : { "t" : 1358377338000, "i" : 1388 }, "optimeDate" : ISODate("2013-01-16T23:02:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2371, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2371, "optime" : { "t" : 1358377333000, "i" : 8505 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:01:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2381, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2369, "optime" : { "t" : 1358377333000, "i" : 8505 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:17Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2368, "optime" : { "t" : 1358377333000, "i" : 8505 }, "optimeDate" : ISODate("2013-01-16T23:02:13Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2368, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2380, "optime" : { "t" : 1358377338000, "i" : 1308 }, "optimeDate" : ISODate("2013-01-16T23:02:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:02:18.276 [conn144] end connection 127.0.0.3:43977 (3 connections now open) m31002| Wed Jan 16 18:02:18.276 [initandlisten] connection accepted from 127.0.0.3:47486 #146 (4 connections now open) m31000| Wed Jan 16 18:02:19.687 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:21.675 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:260375 130ms m31000| Wed Jan 16 18:02:21.688 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 18:02:22.316 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2386, "optime" : { "t" : 1358377342000, "i" : 3094 }, "optimeDate" : ISODate("2013-01-16T23:02:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2375, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2375, "optime" : { "t" : 1358377338000, "i" : 1388 }, "optimeDate" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2385, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2373, "optime" : { "t" : 1358377342000, "i" : 2681 }, "optimeDate" : ISODate("2013-01-16T23:02:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2372, "optime" : { "t" : 1358377338000, "i" : 1388 }, "optimeDate" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2372, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2384, "optime" : { "t" : 1358377342000, "i" : 3094 }, "optimeDate" : ISODate("2013-01-16T23:02:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:23.688 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:25.688 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2390, "optime" : { "t" : 1358377346000, "i" : 4303 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2379, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2379, "optime" : { "t" : 1358377346000, "i" : 707 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2389, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2377, "optime" : { "t" : 1358377346000, "i" : 2469 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2376, "optime" : { "t" : 1358377342000, "i" : 3094 }, "optimeDate" : ISODate("2013-01-16T23:02:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2376, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2388, "optime" : { "t" : 1358377346000, "i" : 3821 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:02:27.317 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:27.689 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:29.683 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:220208 110ms m31000| Wed Jan 16 18:02:29.689 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2394, "optime" : { "t" : 1358377350000, "i" : 5280 }, "optimeDate" : ISODate("2013-01-16T23:02:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2383, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2383, "optime" : { "t" : 1358377346000, "i" : 4303 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2393, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2381, "optime" : { "t" : 1358377350000, "i" : 2370 }, "optimeDate" : ISODate("2013-01-16T23:02:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2380, "optime" : { "t" : 1358377346000, "i" : 4303 }, "optimeDate" : ISODate("2013-01-16T23:02:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2380, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2392, "optime" : { "t" : 1358377350000, "i" : 5280 }, "optimeDate" : ISODate("2013-01-16T23:02:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:02:30.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 18:02:31.689 [conn144] end connection 127.0.0.2:60981 (2 connections now open) m31001| Wed Jan 16 18:02:31.689 [initandlisten] connection accepted from 127.0.0.2:34563 #145 (3 connections now open) m31000| Wed Jan 16 18:02:31.690 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:32.317 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:02:33.568 [conn145] end connection 127.0.0.2:52492 (3 connections now open) m31000| Wed Jan 16 18:02:33.690 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:34.068 [initandlisten] connection accepted from 127.0.0.2:54940 #147 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2398, "optime" : { "t" : 1358377354000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2387, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2387, "optime" : { "t" : 1358377354000, "i" : 438 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:34Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2397, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2385, "optime" : { "t" : 1358377354000, "i" : 2302 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:34Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2384, "optime" : { "t" : 1358377350000, "i" : 5280 }, "optimeDate" : ISODate("2013-01-16T23:02:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2384, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2396, "optime" : { "t" : 1358377354000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:34.913 [conn90] end connection 127.0.0.4:55563 (4 connections now open) m31000| Wed Jan 16 18:02:34.913 [initandlisten] connection accepted from 127.0.0.4:35803 #91 (5 connections now open) m31000| Wed Jan 16 18:02:35.690 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:02:37.318 [conn143] end connection 127.0.0.4:33888 (2 connections now open) m31000| Wed Jan 16 18:02:37.691 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:02:37.818 [initandlisten] connection accepted from 127.0.0.4:54914 #146 (3 connections now open) m31002| Wed Jan 16 18:02:37.818 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2402, "optime" : { "t" : 1358377358000, "i" : 7306 }, "optimeDate" : ISODate("2013-01-16T23:02:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2391, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2391, "optime" : { "t" : 1358377354000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2401, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2389, "optime" : { "t" : 1358377358000, "i" : 2639 }, "optimeDate" : ISODate("2013-01-16T23:02:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2388, "optime" : { "t" : 1358377354000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-16T23:02:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2388, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2400, "optime" : { "t" : 1358377358000, "i" : 7306 }, "optimeDate" : ISODate("2013-01-16T23:02:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:39.691 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:02:40.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:02:41.691 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:42.819 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2406, "optime" : { "t" : 1358377362000, "i" : 8383 }, "optimeDate" : ISODate("2013-01-16T23:02:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2395, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2395, "optime" : { "t" : 1358377358000, "i" : 7306 }, "optimeDate" : ISODate("2013-01-16T23:02:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2405, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2393, "optime" : { "t" : 1358377362000, "i" : 2346 }, "optimeDate" : ISODate("2013-01-16T23:02:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2392, "optime" : { "t" : 1358377362000, "i" : 7995 }, "optimeDate" : ISODate("2013-01-16T23:02:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2392, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2404, "optime" : { "t" : 1358377362000, "i" : 7824 }, "optimeDate" : ISODate("2013-01-16T23:02:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:43.692 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:45.692 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2411, "optime" : { "t" : 1358377367000, "i" : 766 }, "optimeDate" : ISODate("2013-01-16T23:02:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2400, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2400, "optime" : { "t" : 1358377366000, "i" : 5521 }, "optimeDate" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2410, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2398, "optime" : { "t" : 1358377366000, "i" : 2950 }, "optimeDate" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2397, "optime" : { "t" : 1358377366000, "i" : 8709 }, "optimeDate" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2397, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2409, "optime" : { "t" : 1358377367000, "i" : 766 }, "optimeDate" : ISODate("2013-01-16T23:02:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:47.693 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:47.819 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:02:48.281 [conn146] end connection 127.0.0.3:47486 (3 connections now open) m31002| Wed Jan 16 18:02:48.282 [initandlisten] connection accepted from 127.0.0.3:50051 #148 (4 connections now open) m31000| Wed Jan 16 18:02:49.693 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2415, "optime" : { "t" : 1358377371000, "i" : 1415 }, "optimeDate" : ISODate("2013-01-16T23:02:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2404, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2404, "optime" : { "t" : 1358377367000, "i" : 766 }, "optimeDate" : ISODate("2013-01-16T23:02:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2414, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2402, "optime" : { "t" : 1358377370000, "i" : 1640 }, "optimeDate" : ISODate("2013-01-16T23:02:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2401, "optime" : { "t" : 1358377370000, "i" : 7796 }, "optimeDate" : ISODate("2013-01-16T23:02:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2401, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2413, "optime" : { "t" : 1358377371000, "i" : 718 }, "optimeDate" : ISODate("2013-01-16T23:02:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:51.693 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:52.820 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:02:53.694 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2419, "optime" : { "t" : 1358377375000, "i" : 2895 }, "optimeDate" : ISODate("2013-01-16T23:02:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2408, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2408, "optime" : { "t" : 1358377371000, "i" : 1415 }, "optimeDate" : ISODate("2013-01-16T23:02:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2418, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2406, "optime" : { "t" : 1358377374000, "i" : 1088 }, "optimeDate" : ISODate("2013-01-16T23:02:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2405, "optime" : { "t" : 1358377374000, "i" : 6449 }, "optimeDate" : ISODate("2013-01-16T23:02:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2405, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2417, "optime" : { "t" : 1358377375000, "i" : 2895 }, "optimeDate" : ISODate("2013-01-16T23:02:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:55.694 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:02:57.695 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:02:57.821 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2423, "optime" : { "t" : 1358377379000, "i" : 4456 }, "optimeDate" : ISODate("2013-01-16T23:02:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2412, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2412, "optime" : { "t" : 1358377379000, "i" : 215 }, "optimeDate" : ISODate("2013-01-16T23:02:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2422, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2410, "optime" : { "t" : 1358377375000, "i" : 2895 }, "optimeDate" : ISODate("2013-01-16T23:02:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:02:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2409, "optime" : { "t" : 1358377378000, "i" : 5139 }, "optimeDate" : ISODate("2013-01-16T23:02:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2409, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:02:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2421, "optime" : { "t" : 1358377379000, "i" : 3946 }, "optimeDate" : ISODate("2013-01-16T23:02:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:02:59.695 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:01.695 [conn145] end connection 127.0.0.2:34563 (2 connections now open) m31001| Wed Jan 16 18:03:01.695 [initandlisten] connection accepted from 127.0.0.2:34657 #147 (3 connections now open) m31000| Wed Jan 16 18:03:01.696 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:02.821 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:03:02.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2427, "optime" : { "t" : 1358377383000, "i" : 5349 }, "optimeDate" : ISODate("2013-01-16T23:03:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2416, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2416, "optime" : { "t" : 1358377379000, "i" : 4456 }, "optimeDate" : ISODate("2013-01-16T23:02:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2426, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2414, "optime" : { "t" : 1358377379000, "i" : 4456 }, "optimeDate" : ISODate("2013-01-16T23:02:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2413, "optime" : { "t" : 1358377382000, "i" : 4038 }, "optimeDate" : ISODate("2013-01-16T23:03:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2413, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2425, "optime" : { "t" : 1358377383000, "i" : 5349 }, "optimeDate" : ISODate("2013-01-16T23:03:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:03.696 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:03:04.918 [conn91] end connection 127.0.0.4:35803 (4 connections now open) m31000| Wed Jan 16 18:03:04.918 [initandlisten] connection accepted from 127.0.0.4:37096 #92 (5 connections now open) m31000| Wed Jan 16 18:03:05.697 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2431, "optime" : { "t" : 1358377387000, "i" : 6545 }, "optimeDate" : ISODate("2013-01-16T23:03:07Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2420, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2420, "optime" : { "t" : 1358377383000, "i" : 5349 }, "optimeDate" : ISODate("2013-01-16T23:03:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:07Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2430, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2418, "optime" : { "t" : 1358377383000, "i" : 5349 }, "optimeDate" : ISODate("2013-01-16T23:03:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:07Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2417, "optime" : { "t" : 1358377386000, "i" : 3057 }, "optimeDate" : ISODate("2013-01-16T23:03:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2417, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:06Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2429, "optime" : { "t" : 1358377387000, "i" : 5856 }, "optimeDate" : ISODate("2013-01-16T23:03:07Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:07.697 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:07.822 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:09.698 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:11.573 [conn147] end connection 127.0.0.2:54940 (3 connections now open) m31000| Wed Jan 16 18:03:11.698 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2435, "optime" : { "t" : 1358377391000, "i" : 7032 }, "optimeDate" : ISODate("2013-01-16T23:03:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2424, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2424, "optime" : { "t" : 1358377387000, "i" : 6545 }, "optimeDate" : ISODate("2013-01-16T23:03:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2434, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2422, "optime" : { "t" : 1358377387000, "i" : 6545 }, "optimeDate" : ISODate("2013-01-16T23:03:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2421, "optime" : { "t" : 1358377390000, "i" : 1820 }, "optimeDate" : ISODate("2013-01-16T23:03:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2421, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2433, "optime" : { "t" : 1358377391000, "i" : 7032 }, "optimeDate" : ISODate("2013-01-16T23:03:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:03:12.073 [initandlisten] connection accepted from 127.0.0.2:59380 #149 (4 connections now open) m31002| Wed Jan 16 18:03:12.823 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:03:12.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:03:13.698 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:15.323 [conn146] end connection 127.0.0.4:54914 (2 connections now open) m31000| Wed Jan 16 18:03:15.699 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:15.823 [initandlisten] connection accepted from 127.0.0.4:33618 #148 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2440, "optime" : { "t" : 1358377396000, "i" : 441 }, "optimeDate" : ISODate("2013-01-16T23:03:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2429, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2429, "optime" : { "t" : 1358377391000, "i" : 7032 }, "optimeDate" : ISODate("2013-01-16T23:03:11Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:14Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2439, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2427, "optime" : { "t" : 1358377391000, "i" : 7032 }, "optimeDate" : ISODate("2013-01-16T23:03:11Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:15Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2426, "optime" : { "t" : 1358377394000, "i" : 58 }, "optimeDate" : ISODate("2013-01-16T23:03:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2426, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2438, "optime" : { "t" : 1358377396000, "i" : 441 }, "optimeDate" : ISODate("2013-01-16T23:03:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:17.699 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:18.287 [conn148] end connection 127.0.0.3:50051 (3 connections now open) m31002| Wed Jan 16 18:03:18.287 [initandlisten] connection accepted from 127.0.0.3:37396 #150 (4 connections now open) m31002| Wed Jan 16 18:03:18.324 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:19.699 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2444, "optime" : { "t" : 1358377400000, "i" : 2292 }, "optimeDate" : ISODate("2013-01-16T23:03:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2433, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2433, "optime" : { "t" : 1358377399000, "i" : 4418 }, "optimeDate" : ISODate("2013-01-16T23:03:19Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:02:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2443, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2431, "optime" : { "t" : 1358377396000, "i" : 441 }, "optimeDate" : ISODate("2013-01-16T23:03:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:18Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2430, "optime" : { "t" : 1358377396000, "i" : 441 }, "optimeDate" : ISODate("2013-01-16T23:03:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2430, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2442, "optime" : { "t" : 1358377400000, "i" : 2292 }, "optimeDate" : ISODate("2013-01-16T23:03:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:21.700 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:23.324 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:23.700 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2448, "optime" : { "t" : 1358377404000, "i" : 3158 }, "optimeDate" : ISODate("2013-01-16T23:03:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2437, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2437, "optime" : { "t" : 1358377400000, "i" : 2292 }, "optimeDate" : ISODate("2013-01-16T23:03:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2447, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2435, "optime" : { "t" : 1358377404000, "i" : 2194 }, "optimeDate" : ISODate("2013-01-16T23:03:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2434, "optime" : { "t" : 1358377400000, "i" : 2292 }, "optimeDate" : ISODate("2013-01-16T23:03:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2434, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2446, "optime" : { "t" : 1358377404000, "i" : 2649 }, "optimeDate" : ISODate("2013-01-16T23:03:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:25.700 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:03:27.701 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:28.325 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2452, "optime" : { "t" : 1358377408000, "i" : 5152 }, "optimeDate" : ISODate("2013-01-16T23:03:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2441, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2441, "optime" : { "t" : 1358377404000, "i" : 3158 }, "optimeDate" : ISODate("2013-01-16T23:03:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2451, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2439, "optime" : { "t" : 1358377408000, "i" : 2531 }, "optimeDate" : ISODate("2013-01-16T23:03:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:28Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2438, "optime" : { "t" : 1358377404000, "i" : 3158 }, "optimeDate" : ISODate("2013-01-16T23:03:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2438, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2450, "optime" : { "t" : 1358377408000, "i" : 4919 }, "optimeDate" : ISODate("2013-01-16T23:03:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:29.701 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:31.701 [conn147] end connection 127.0.0.2:34657 (2 connections now open) m31001| Wed Jan 16 18:03:31.701 [initandlisten] connection accepted from 127.0.0.2:56767 #149 (3 connections now open) m31000| Wed Jan 16 18:03:31.701 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2456, "optime" : { "t" : 1358377412000, "i" : 6857 }, "optimeDate" : ISODate("2013-01-16T23:03:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2445, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2445, "optime" : { "t" : 1358377412000, "i" : 646 }, "optimeDate" : ISODate("2013-01-16T23:03:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2455, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2443, "optime" : { "t" : 1358377412000, "i" : 2338 }, "optimeDate" : ISODate("2013-01-16T23:03:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2442, "optime" : { "t" : 1358377408000, "i" : 5152 }, "optimeDate" : ISODate("2013-01-16T23:03:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2442, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2454, "optime" : { "t" : 1358377412000, "i" : 6608 }, "optimeDate" : ISODate("2013-01-16T23:03:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:03:33.326 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:33.702 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:03:34.924 [conn92] end connection 127.0.0.4:37096 (4 connections now open) m31000| Wed Jan 16 18:03:34.924 [initandlisten] connection accepted from 127.0.0.4:60035 #93 (5 connections now open) m31000| Wed Jan 16 18:03:35.702 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2460, "optime" : { "t" : 1358377416000, "i" : 8287 }, "optimeDate" : ISODate("2013-01-16T23:03:36Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2449, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2449, "optime" : { "t" : 1358377412000, "i" : 6857 }, "optimeDate" : ISODate("2013-01-16T23:03:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:36Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2459, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2447, "optime" : { "t" : 1358377416000, "i" : 2242 }, "optimeDate" : ISODate("2013-01-16T23:03:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:36Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2446, "optime" : { "t" : 1358377416000, "i" : 7593 }, "optimeDate" : ISODate("2013-01-16T23:03:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2446, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2458, "optime" : { "t" : 1358377416000, "i" : 8287 }, "optimeDate" : ISODate("2013-01-16T23:03:36Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:37.703 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:38.326 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:39.703 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2465, "optime" : { "t" : 1358377421000, "i" : 566 }, "optimeDate" : ISODate("2013-01-16T23:03:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2454, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2454, "optime" : { "t" : 1358377416000, "i" : 8287 }, "optimeDate" : ISODate("2013-01-16T23:03:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2464, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2452, "optime" : { "t" : 1358377420000, "i" : 2489 }, "optimeDate" : ISODate("2013-01-16T23:03:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2451, "optime" : { "t" : 1358377420000, "i" : 8646 }, "optimeDate" : ISODate("2013-01-16T23:03:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2451, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2463, "optime" : { "t" : 1358377421000, "i" : 37 }, "optimeDate" : ISODate("2013-01-16T23:03:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:41.704 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:43.327 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:43.704 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:44.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2469, "optime" : { "t" : 1358377425000, "i" : 1514 }, "optimeDate" : ISODate("2013-01-16T23:03:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2458, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2458, "optime" : { "t" : 1358377424000, "i" : 4317 }, "optimeDate" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2468, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2456, "optime" : { "t" : 1358377424000, "i" : 1520 }, "optimeDate" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2455, "optime" : { "t" : 1358377424000, "i" : 7434 }, "optimeDate" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2455, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2467, "optime" : { "t" : 1358377425000, "i" : 1219 }, "optimeDate" : ISODate("2013-01-16T23:03:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:45.704 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:03:47.705 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:48.292 [conn150] end connection 127.0.0.3:37396 (3 connections now open) m31002| Wed Jan 16 18:03:48.293 [initandlisten] connection accepted from 127.0.0.3:40019 #151 (5 connections now open) m31002| Wed Jan 16 18:03:48.328 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2473, "optime" : { "t" : 1358377429000, "i" : 2705 }, "optimeDate" : ISODate("2013-01-16T23:03:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2462, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2462, "optime" : { "t" : 1358377425000, "i" : 1514 }, "optimeDate" : ISODate("2013-01-16T23:03:45Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2472, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2460, "optime" : { "t" : 1358377428000, "i" : 637 }, "optimeDate" : ISODate("2013-01-16T23:03:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2459, "optime" : { "t" : 1358377428000, "i" : 6616 }, "optimeDate" : ISODate("2013-01-16T23:03:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2459, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2471, "optime" : { "t" : 1358377429000, "i" : 2705 }, "optimeDate" : ISODate("2013-01-16T23:03:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:03:49.579 [conn149] end connection 127.0.0.2:59380 (3 connections now open) m31000| Wed Jan 16 18:03:49.705 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:50.079 [initandlisten] connection accepted from 127.0.0.2:44239 #152 (4 connections now open) m31000| Wed Jan 16 18:03:51.706 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:53.328 [conn148] end connection 127.0.0.4:33618 (2 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2477, "optime" : { "t" : 1358377433000, "i" : 4391 }, "optimeDate" : ISODate("2013-01-16T23:03:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2466, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2466, "optime" : { "t" : 1358377432000, "i" : 2300 }, "optimeDate" : ISODate("2013-01-16T23:03:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2476, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2464, "optime" : { "t" : 1358377429000, "i" : 2705 }, "optimeDate" : ISODate("2013-01-16T23:03:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2463, "optime" : { "t" : 1358377432000, "i" : 5363 }, "optimeDate" : ISODate("2013-01-16T23:03:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2463, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2475, "optime" : { "t" : 1358377433000, "i" : 4391 }, "optimeDate" : ISODate("2013-01-16T23:03:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:53.706 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:03:53.828 [initandlisten] connection accepted from 127.0.0.4:48742 #150 (3 connections now open) m31002| Wed Jan 16 18:03:53.829 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:55.706 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2481, "optime" : { "t" : 1358377437000, "i" : 5755 }, "optimeDate" : ISODate("2013-01-16T23:03:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2470, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2470, "optime" : { "t" : 1358377437000, "i" : 5130 }, "optimeDate" : ISODate("2013-01-16T23:03:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2480, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2468, "optime" : { "t" : 1358377433000, "i" : 4391 }, "optimeDate" : ISODate("2013-01-16T23:03:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:03:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2467, "optime" : { "t" : 1358377436000, "i" : 3842 }, "optimeDate" : ISODate("2013-01-16T23:03:56Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:03:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2467, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2479, "optime" : { "t" : 1358377437000, "i" : 5755 }, "optimeDate" : ISODate("2013-01-16T23:03:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:03:57.707 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:03:58.829 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:03:59.707 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:04:01.707 [conn149] end connection 127.0.0.2:56767 (2 connections now open) m31001| Wed Jan 16 18:04:01.707 [initandlisten] connection accepted from 127.0.0.2:39764 #151 (3 connections now open) m31000| Wed Jan 16 18:04:01.708 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2485, "optime" : { "t" : 1358377441000, "i" : 6918 }, "optimeDate" : ISODate("2013-01-16T23:04:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2474, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2474, "optime" : { "t" : 1358377437000, "i" : 5755 }, "optimeDate" : ISODate("2013-01-16T23:03:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2484, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2472, "optime" : { "t" : 1358377437000, "i" : 5755 }, "optimeDate" : ISODate("2013-01-16T23:03:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:01Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2471, "optime" : { "t" : 1358377440000, "i" : 2562 }, "optimeDate" : ISODate("2013-01-16T23:04:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2471, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2483, "optime" : { "t" : 1358377441000, "i" : 6918 }, "optimeDate" : ISODate("2013-01-16T23:04:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:03.708 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:03.830 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:04.928 [conn93] end connection 127.0.0.4:60035 (4 connections now open) m31000| Wed Jan 16 18:04:04.929 [initandlisten] connection accepted from 127.0.0.4:60561 #94 (5 connections now open) m31000| Wed Jan 16 18:04:05.708 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2489, "optime" : { "t" : 1358377445000, "i" : 8077 }, "optimeDate" : ISODate("2013-01-16T23:04:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2478, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2478, "optime" : { "t" : 1358377445000, "i" : 485 }, "optimeDate" : ISODate("2013-01-16T23:04:05Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2488, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2476, "optime" : { "t" : 1358377441000, "i" : 6918 }, "optimeDate" : ISODate("2013-01-16T23:04:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2475, "optime" : { "t" : 1358377444000, "i" : 1342 }, "optimeDate" : ISODate("2013-01-16T23:04:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2475, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:04Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2487, "optime" : { "t" : 1358377445000, "i" : 8077 }, "optimeDate" : ISODate("2013-01-16T23:04:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:04:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:04:07.709 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:08.831 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:09.709 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2494, "optime" : { "t" : 1358377450000, "i" : 821 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2483, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2483, "optime" : { "t" : 1358377450000, "i" : 694 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2493, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2481, "optime" : { "t" : 1358377445000, "i" : 8077 }, "optimeDate" : ISODate("2013-01-16T23:04:05Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:08Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2480, "optime" : { "t" : 1358377448000, "i" : 43 }, "optimeDate" : ISODate("2013-01-16T23:04:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2480, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2492, "optime" : { "t" : 1358377450000, "i" : 694 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:11.709 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:13.710 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:13.831 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2498, "optime" : { "t" : 1358377454000, "i" : 2936 }, "optimeDate" : ISODate("2013-01-16T23:04:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2487, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2487, "optime" : { "t" : 1358377450000, "i" : 821 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:12Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2497, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2485, "optime" : { "t" : 1358377450000, "i" : 821 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2484, "optime" : { "t" : 1358377450000, "i" : 821 }, "optimeDate" : ISODate("2013-01-16T23:04:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2484, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2496, "optime" : { "t" : 1358377454000, "i" : 2874 }, "optimeDate" : ISODate("2013-01-16T23:04:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:15.710 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:04:16.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:04:17.710 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:18.298 [conn151] end connection 127.0.0.3:40019 (3 connections now open) m31002| Wed Jan 16 18:04:18.298 [initandlisten] connection accepted from 127.0.0.3:35375 #153 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2502, "optime" : { "t" : 1358377458000, "i" : 3749 }, "optimeDate" : ISODate("2013-01-16T23:04:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2491, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2491, "optime" : { "t" : 1358377457000, "i" : 2190 }, "optimeDate" : ISODate("2013-01-16T23:04:17Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2501, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2489, "optime" : { "t" : 1358377458000, "i" : 2075 }, "optimeDate" : ISODate("2013-01-16T23:04:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2488, "optime" : { "t" : 1358377454000, "i" : 2936 }, "optimeDate" : ISODate("2013-01-16T23:04:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2488, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2500, "optime" : { "t" : 1358377458000, "i" : 3749 }, "optimeDate" : ISODate("2013-01-16T23:04:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:04:18.832 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:19.711 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:21.711 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2506, "optime" : { "t" : 1358377462000, "i" : 5408 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2495, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2495, "optime" : { "t" : 1358377462000, "i" : 5270 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2505, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2493, "optime" : { "t" : 1358377462000, "i" : 2984 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2492, "optime" : { "t" : 1358377458000, "i" : 3749 }, "optimeDate" : ISODate("2013-01-16T23:04:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:22Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2492, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2504, "optime" : { "t" : 1358377462000, "i" : 5408 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:23.711 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:23.833 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:25.712 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2510, "optime" : { "t" : 1358377466000, "i" : 6444 }, "optimeDate" : ISODate("2013-01-16T23:04:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2499, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2499, "optime" : { "t" : 1358377462000, "i" : 5408 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2509, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2497, "optime" : { "t" : 1358377466000, "i" : 2701 }, "optimeDate" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:26Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2496, "optime" : { "t" : 1358377462000, "i" : 5408 }, "optimeDate" : ISODate("2013-01-16T23:04:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2496, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2508, "optime" : { "t" : 1358377466000, "i" : 6226 }, "optimeDate" : ISODate("2013-01-16T23:04:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:04:27.584 [conn152] end connection 127.0.0.2:44239 (3 connections now open) m31000| Wed Jan 16 18:04:27.712 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:28.084 [initandlisten] connection accepted from 127.0.0.2:52785 #154 (4 connections now open) m31002| Wed Jan 16 18:04:28.833 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:29.712 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2514, "optime" : { "t" : 1358377470000, "i" : 8296 }, "optimeDate" : ISODate("2013-01-16T23:04:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2503, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2503, "optime" : { "t" : 1358377470000, "i" : 5239 }, "optimeDate" : ISODate("2013-01-16T23:04:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2513, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2501, "optime" : { "t" : 1358377470000, "i" : 2420 }, "optimeDate" : ISODate("2013-01-16T23:04:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2500, "optime" : { "t" : 1358377466000, "i" : 6444 }, "optimeDate" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2500, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2512, "optime" : { "t" : 1358377470000, "i" : 8296 }, "optimeDate" : ISODate("2013-01-16T23:04:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:04:31.333 [conn150] end connection 127.0.0.4:48742 (2 connections now open) m31001| Wed Jan 16 18:04:31.713 [conn151] end connection 127.0.0.2:39764 (1 connection now open) m31001| Wed Jan 16 18:04:31.713 [initandlisten] connection accepted from 127.0.0.2:55478 #152 (2 connections now open) m31000| Wed Jan 16 18:04:31.713 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:04:31.833 [initandlisten] connection accepted from 127.0.0.4:41844 #153 (3 connections now open) m31000| Wed Jan 16 18:04:33.714 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:34.334 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:34.934 [conn94] end connection 127.0.0.4:60561 (4 connections now open) m31000| Wed Jan 16 18:04:34.934 [initandlisten] connection accepted from 127.0.0.4:57070 #95 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2519, "optime" : { "t" : 1358377475000, "i" : 367 }, "optimeDate" : ISODate("2013-01-16T23:04:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2508, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2508, "optime" : { "t" : 1358377470000, "i" : 8296 }, "optimeDate" : ISODate("2013-01-16T23:04:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2518, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2506, "optime" : { "t" : 1358377474000, "i" : 2291 }, "optimeDate" : ISODate("2013-01-16T23:04:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2505, "optime" : { "t" : 1358377474000, "i" : 8469 }, "optimeDate" : ISODate("2013-01-16T23:04:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2505, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2517, "optime" : { "t" : 1358377475000, "i" : 367 }, "optimeDate" : ISODate("2013-01-16T23:04:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:35.714 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:37.715 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:38.212 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:256045 128ms m31000| Wed Jan 16 18:04:38.412 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:261298 130ms m31001| Wed Jan 16 18:04:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2523, "optime" : { "t" : 1358377479000, "i" : 671 }, "optimeDate" : ISODate("2013-01-16T23:04:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2512, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2512, "optime" : { "t" : 1358377478000, "i" : 42 }, "optimeDate" : ISODate("2013-01-16T23:04:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2522, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2510, "optime" : { "t" : 1358377478000, "i" : 2231 }, "optimeDate" : ISODate("2013-01-16T23:04:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2509, "optime" : { "t" : 1358377478000, "i" : 8489 }, "optimeDate" : ISODate("2013-01-16T23:04:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2509, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2521, "optime" : { "t" : 1358377479000, "i" : 596 }, "optimeDate" : ISODate("2013-01-16T23:04:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:04:39.335 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:39.715 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:41.715 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2527, "optime" : { "t" : 1358377483000, "i" : 2102 }, "optimeDate" : ISODate("2013-01-16T23:04:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2516, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2516, "optime" : { "t" : 1358377483000, "i" : 955 }, "optimeDate" : ISODate("2013-01-16T23:04:43Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2526, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2514, "optime" : { "t" : 1358377482000, "i" : 1431 }, "optimeDate" : ISODate("2013-01-16T23:04:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2513, "optime" : { "t" : 1358377482000, "i" : 7575 }, "optimeDate" : ISODate("2013-01-16T23:04:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2513, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2525, "optime" : { "t" : 1358377483000, "i" : 2102 }, "optimeDate" : ISODate("2013-01-16T23:04:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:43.716 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:44.335 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:45.716 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2531, "optime" : { "t" : 1358377487000, "i" : 3168 }, "optimeDate" : ISODate("2013-01-16T23:04:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2520, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2520, "optime" : { "t" : 1358377483000, "i" : 2102 }, "optimeDate" : ISODate("2013-01-16T23:04:43Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2530, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2518, "optime" : { "t" : 1358377486000, "i" : 441 }, "optimeDate" : ISODate("2013-01-16T23:04:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:46Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2517, "optime" : { "t" : 1358377486000, "i" : 6299 }, "optimeDate" : ISODate("2013-01-16T23:04:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2517, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2529, "optime" : { "t" : 1358377487000, "i" : 2866 }, "optimeDate" : ISODate("2013-01-16T23:04:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:47.717 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:48.303 [conn153] end connection 127.0.0.3:35375 (3 connections now open) m31002| Wed Jan 16 18:04:48.303 [initandlisten] connection accepted from 127.0.0.3:36372 #155 (4 connections now open) m31001| Wed Jan 16 18:04:48.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 18:04:49.336 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:49.717 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2535, "optime" : { "t" : 1358377491000, "i" : 4329 }, "optimeDate" : ISODate("2013-01-16T23:04:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2524, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2524, "optime" : { "t" : 1358377490000, "i" : 1386 }, "optimeDate" : ISODate("2013-01-16T23:04:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2534, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2522, "optime" : { "t" : 1358377487000, "i" : 3168 }, "optimeDate" : ISODate("2013-01-16T23:04:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2521, "optime" : { "t" : 1358377490000, "i" : 4671 }, "optimeDate" : ISODate("2013-01-16T23:04:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2521, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2533, "optime" : { "t" : 1358377491000, "i" : 4329 }, "optimeDate" : ISODate("2013-01-16T23:04:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:51.717 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:53.718 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:54.337 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2539, "optime" : { "t" : 1358377495000, "i" : 5739 }, "optimeDate" : ISODate("2013-01-16T23:04:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2528, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2528, "optime" : { "t" : 1358377495000, "i" : 4711 }, "optimeDate" : ISODate("2013-01-16T23:04:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2538, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2526, "optime" : { "t" : 1358377491000, "i" : 4329 }, "optimeDate" : ISODate("2013-01-16T23:04:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2525, "optime" : { "t" : 1358377494000, "i" : 3388 }, "optimeDate" : ISODate("2013-01-16T23:04:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2525, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2537, "optime" : { "t" : 1358377495000, "i" : 5739 }, "optimeDate" : ISODate("2013-01-16T23:04:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:04:55.718 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:04:57.719 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:04:59.337 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:04:59.719 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2543, "optime" : { "t" : 1358377499000, "i" : 6571 }, "optimeDate" : ISODate("2013-01-16T23:04:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2532, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2532, "optime" : { "t" : 1358377495000, "i" : 5739 }, "optimeDate" : ISODate("2013-01-16T23:04:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2542, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2530, "optime" : { "t" : 1358377495000, "i" : 5739 }, "optimeDate" : ISODate("2013-01-16T23:04:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:04:59Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:04:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2529, "optime" : { "t" : 1358377498000, "i" : 2833 }, "optimeDate" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2529, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2541, "optime" : { "t" : 1358377499000, "i" : 6571 }, "optimeDate" : ISODate("2013-01-16T23:04:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:05:01.719 [conn152] end connection 127.0.0.2:55478 (2 connections now open) m31001| Wed Jan 16 18:05:01.719 [initandlisten] connection accepted from 127.0.0.2:48192 #154 (3 connections now open) m31000| Wed Jan 16 18:05:01.720 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:03.720 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2547, "optime" : { "t" : 1358377503000, "i" : 8275 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2536, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2536, "optime" : { "t" : 1358377503000, "i" : 283 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2546, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2534, "optime" : { "t" : 1358377499000, "i" : 6571 }, "optimeDate" : ISODate("2013-01-16T23:04:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2533, "optime" : { "t" : 1358377502000, "i" : 917 }, "optimeDate" : ISODate("2013-01-16T23:05:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2533, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:02Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2545, "optime" : { "t" : 1358377503000, "i" : 8210 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:05:04.338 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:04.939 [conn95] end connection 127.0.0.4:57070 (4 connections now open) m31000| Wed Jan 16 18:05:04.939 [initandlisten] connection accepted from 127.0.0.4:51539 #96 (5 connections now open) m31002| Wed Jan 16 18:05:05.589 [conn154] end connection 127.0.0.2:52785 (3 connections now open) m31000| Wed Jan 16 18:05:05.720 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:06.089 [initandlisten] connection accepted from 127.0.0.2:55827 #156 (4 connections now open) m31000| Wed Jan 16 18:05:07.721 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2552, "optime" : { "t" : 1358377508000, "i" : 910 }, "optimeDate" : ISODate("2013-01-16T23:05:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2541, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2541, "optime" : { "t" : 1358377503000, "i" : 8275 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2551, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2539, "optime" : { "t" : 1358377503000, "i" : 8275 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2538, "optime" : { "t" : 1358377503000, "i" : 8275 }, "optimeDate" : ISODate("2013-01-16T23:05:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2538, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2550, "optime" : { "t" : 1358377508000, "i" : 910 }, "optimeDate" : ISODate("2013-01-16T23:05:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:05:09.339 [conn153] end connection 127.0.0.4:41844 (2 connections now open) m31000| Wed Jan 16 18:05:09.721 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:05:09.839 [initandlisten] connection accepted from 127.0.0.4:50477 #155 (3 connections now open) m31002| Wed Jan 16 18:05:09.839 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:05:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:05:11.721 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2556, "optime" : { "t" : 1358377512000, "i" : 2707 }, "optimeDate" : ISODate("2013-01-16T23:05:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2545, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2545, "optime" : { "t" : 1358377508000, "i" : 910 }, "optimeDate" : ISODate("2013-01-16T23:05:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2555, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2543, "optime" : { "t" : 1358377508000, "i" : 910 }, "optimeDate" : ISODate("2013-01-16T23:05:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2542, "optime" : { "t" : 1358377508000, "i" : 910 }, "optimeDate" : ISODate("2013-01-16T23:05:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2542, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:10Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2554, "optime" : { "t" : 1358377512000, "i" : 2537 }, "optimeDate" : ISODate("2013-01-16T23:05:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:13.722 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:14.840 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:15.722 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2560, "optime" : { "t" : 1358377516000, "i" : 3877 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2549, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2549, "optime" : { "t" : 1358377516000, "i" : 777 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2559, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2547, "optime" : { "t" : 1358377516000, "i" : 2498 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2546, "optime" : { "t" : 1358377512000, "i" : 2707 }, "optimeDate" : ISODate("2013-01-16T23:05:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2546, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2558, "optime" : { "t" : 1358377516000, "i" : 3877 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:17.722 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:18.309 [conn155] end connection 127.0.0.3:36372 (3 connections now open) m31002| Wed Jan 16 18:05:18.309 [initandlisten] connection accepted from 127.0.0.3:52020 #157 (4 connections now open) m31000| Wed Jan 16 18:05:19.723 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:19.840 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2564, "optime" : { "t" : 1358377520000, "i" : 5310 }, "optimeDate" : ISODate("2013-01-16T23:05:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2553, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2553, "optime" : { "t" : 1358377516000, "i" : 3877 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:18Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2551, "optime" : { "t" : 1358377520000, "i" : 2512 }, "optimeDate" : ISODate("2013-01-16T23:05:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2550, "optime" : { "t" : 1358377516000, "i" : 3877 }, "optimeDate" : ISODate("2013-01-16T23:05:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2550, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2562, "optime" : { "t" : 1358377520000, "i" : 5310 }, "optimeDate" : ISODate("2013-01-16T23:05:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:05:20.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:05:21.723 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:23.723 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:24.642 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:410211 205ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2568, "optime" : { "t" : 1358377524000, "i" : 6873 }, "optimeDate" : ISODate("2013-01-16T23:05:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2557, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2557, "optime" : { "t" : 1358377520000, "i" : 5310 }, "optimeDate" : ISODate("2013-01-16T23:05:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2567, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2555, "optime" : { "t" : 1358377524000, "i" : 2749 }, "optimeDate" : ISODate("2013-01-16T23:05:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2554, "optime" : { "t" : 1358377520000, "i" : 5310 }, "optimeDate" : ISODate("2013-01-16T23:05:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2554, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2566, "optime" : { "t" : 1358377524000, "i" : 5703 }, "optimeDate" : ISODate("2013-01-16T23:05:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:05:24.841 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:25.724 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:27.724 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2572, "optime" : { "t" : 1358377528000, "i" : 8573 }, "optimeDate" : ISODate("2013-01-16T23:05:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2561, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2561, "optime" : { "t" : 1358377528000, "i" : 5765 }, "optimeDate" : ISODate("2013-01-16T23:05:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:04:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2571, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2559, "optime" : { "t" : 1358377528000, "i" : 3078 }, "optimeDate" : ISODate("2013-01-16T23:05:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2558, "optime" : { "t" : 1358377524000, "i" : 6873 }, "optimeDate" : ISODate("2013-01-16T23:05:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2558, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2570, "optime" : { "t" : 1358377528000, "i" : 8573 }, "optimeDate" : ISODate("2013-01-16T23:05:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:29.725 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:29.842 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:05:31.725 [conn154] end connection 127.0.0.2:48192 (2 connections now open) m31001| Wed Jan 16 18:05:31.725 [initandlisten] connection accepted from 127.0.0.2:35993 #156 (3 connections now open) m31000| Wed Jan 16 18:05:31.725 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2577, "optime" : { "t" : 1358377533000, "i" : 880 }, "optimeDate" : ISODate("2013-01-16T23:05:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2566, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2566, "optime" : { "t" : 1358377528000, "i" : 8573 }, "optimeDate" : ISODate("2013-01-16T23:05:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2576, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2564, "optime" : { "t" : 1358377532000, "i" : 2944 }, "optimeDate" : ISODate("2013-01-16T23:05:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2563, "optime" : { "t" : 1358377532000, "i" : 8300 }, "optimeDate" : ISODate("2013-01-16T23:05:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2563, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2575, "optime" : { "t" : 1358377533000, "i" : 880 }, "optimeDate" : ISODate("2013-01-16T23:05:33Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:33.726 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:34.842 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:34.945 [conn96] end connection 127.0.0.4:51539 (4 connections now open) m31000| Wed Jan 16 18:05:34.945 [initandlisten] connection accepted from 127.0.0.4:49945 #97 (5 connections now open) m31000| Wed Jan 16 18:05:35.726 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:36.951 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293252 146ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2581, "optime" : { "t" : 1358377537000, "i" : 2121 }, "optimeDate" : ISODate("2013-01-16T23:05:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2570, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2570, "optime" : { "t" : 1358377536000, "i" : 183 }, "optimeDate" : ISODate("2013-01-16T23:05:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2580, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2568, "optime" : { "t" : 1358377536000, "i" : 2092 }, "optimeDate" : ISODate("2013-01-16T23:05:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2567, "optime" : { "t" : 1358377536000, "i" : 6823 }, "optimeDate" : ISODate("2013-01-16T23:05:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2567, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2579, "optime" : { "t" : 1358377537000, "i" : 1347 }, "optimeDate" : ISODate("2013-01-16T23:05:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:37.727 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:39.727 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:39.843 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2585, "optime" : { "t" : 1358377541000, "i" : 2417 }, "optimeDate" : ISODate("2013-01-16T23:05:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2574, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2574, "optime" : { "t" : 1358377541000, "i" : 682 }, "optimeDate" : ISODate("2013-01-16T23:05:41Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2584, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2572, "optime" : { "t" : 1358377540000, "i" : 1447 }, "optimeDate" : ISODate("2013-01-16T23:05:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2571, "optime" : { "t" : 1358377540000, "i" : 7271 }, "optimeDate" : ISODate("2013-01-16T23:05:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2571, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2583, "optime" : { "t" : 1358377541000, "i" : 2417 }, "optimeDate" : ISODate("2013-01-16T23:05:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:41.727 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:05:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 18:05:43.594 [conn156] end connection 127.0.0.2:55827 (3 connections now open) m31000| Wed Jan 16 18:05:43.728 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:44.094 [initandlisten] connection accepted from 127.0.0.2:54009 #158 (4 connections now open) m31002| Wed Jan 16 18:05:44.844 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2589, "optime" : { "t" : 1358377545000, "i" : 3558 }, "optimeDate" : ISODate("2013-01-16T23:05:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2578, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2578, "optime" : { "t" : 1358377541000, "i" : 2417 }, "optimeDate" : ISODate("2013-01-16T23:05:41Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2588, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2576, "optime" : { "t" : 1358377544000, "i" : 207 }, "optimeDate" : ISODate("2013-01-16T23:05:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2575, "optime" : { "t" : 1358377544000, "i" : 6158 }, "optimeDate" : ISODate("2013-01-16T23:05:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2575, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2587, "optime" : { "t" : 1358377545000, "i" : 3504 }, "optimeDate" : ISODate("2013-01-16T23:05:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:45.728 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:05:47.344 [conn155] end connection 127.0.0.4:50477 (2 connections now open) m31000| Wed Jan 16 18:05:47.729 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:05:47.844 [initandlisten] connection accepted from 127.0.0.4:56803 #157 (3 connections now open) m31002| Wed Jan 16 18:05:48.314 [conn157] end connection 127.0.0.3:52020 (3 connections now open) m31002| Wed Jan 16 18:05:48.314 [initandlisten] connection accepted from 127.0.0.3:59792 #159 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2593, "optime" : { "t" : 1358377549000, "i" : 5101 }, "optimeDate" : ISODate("2013-01-16T23:05:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2582, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2582, "optime" : { "t" : 1358377549000, "i" : 963 }, "optimeDate" : ISODate("2013-01-16T23:05:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2592, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2580, "optime" : { "t" : 1358377545000, "i" : 3558 }, "optimeDate" : ISODate("2013-01-16T23:05:45Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2579, "optime" : { "t" : 1358377548000, "i" : 4655 }, "optimeDate" : ISODate("2013-01-16T23:05:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2579, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:48Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2591, "optime" : { "t" : 1358377549000, "i" : 5098 }, "optimeDate" : ISODate("2013-01-16T23:05:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:49.729 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:50.345 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:51.729 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:05:52.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2597, "optime" : { "t" : 1358377553000, "i" : 6561 }, "optimeDate" : ISODate("2013-01-16T23:05:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2586, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2586, "optime" : { "t" : 1358377549000, "i" : 5101 }, "optimeDate" : ISODate("2013-01-16T23:05:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:52Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2596, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2584, "optime" : { "t" : 1358377549000, "i" : 5101 }, "optimeDate" : ISODate("2013-01-16T23:05:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:52Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2583, "optime" : { "t" : 1358377552000, "i" : 3159 }, "optimeDate" : ISODate("2013-01-16T23:05:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2583, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2595, "optime" : { "t" : 1358377553000, "i" : 6408 }, "optimeDate" : ISODate("2013-01-16T23:05:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:53.730 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:05:55.345 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:05:55.730 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:05:57.730 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2601, "optime" : { "t" : 1358377557000, "i" : 7472 }, "optimeDate" : ISODate("2013-01-16T23:05:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2590, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2590, "optime" : { "t" : 1358377553000, "i" : 6561 }, "optimeDate" : ISODate("2013-01-16T23:05:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2600, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2588, "optime" : { "t" : 1358377553000, "i" : 6561 }, "optimeDate" : ISODate("2013-01-16T23:05:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:05:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:05:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2587, "optime" : { "t" : 1358377556000, "i" : 1623 }, "optimeDate" : ISODate("2013-01-16T23:05:56Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2587, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2599, "optime" : { "t" : 1358377557000, "i" : 7357 }, "optimeDate" : ISODate("2013-01-16T23:05:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:05:59.731 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:00.346 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:06:01.731 [conn156] end connection 127.0.0.2:35993 (2 connections now open) m31001| Wed Jan 16 18:06:01.731 [initandlisten] connection accepted from 127.0.0.2:38974 #158 (3 connections now open) m31000| Wed Jan 16 18:06:01.731 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2606, "optime" : { "t" : 1358377562000, "i" : 406 }, "optimeDate" : ISODate("2013-01-16T23:06:02Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2595, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2595, "optime" : { "t" : 1358377561000, "i" : 4984 }, "optimeDate" : ISODate("2013-01-16T23:06:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:02Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2605, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2593, "optime" : { "t" : 1358377557000, "i" : 7472 }, "optimeDate" : ISODate("2013-01-16T23:05:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:02Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2592, "optime" : { "t" : 1358377560000, "i" : 423 }, "optimeDate" : ISODate("2013-01-16T23:06:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2592, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:05:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2604, "optime" : { "t" : 1358377562000, "i" : 247 }, "optimeDate" : ISODate("2013-01-16T23:06:02Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:03.732 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:04.950 [conn97] end connection 127.0.0.4:49945 (4 connections now open) m31000| Wed Jan 16 18:06:04.950 [initandlisten] connection accepted from 127.0.0.4:35774 #98 (5 connections now open) m31002| Wed Jan 16 18:06:05.347 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:06:05.732 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2610, "optime" : { "t" : 1358377566000, "i" : 2244 }, "optimeDate" : ISODate("2013-01-16T23:06:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2599, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2599, "optime" : { "t" : 1358377562000, "i" : 406 }, "optimeDate" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:04Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2609, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2597, "optime" : { "t" : 1358377562000, "i" : 406 }, "optimeDate" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:05Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2596, "optime" : { "t" : 1358377562000, "i" : 406 }, "optimeDate" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2596, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2608, "optime" : { "t" : 1358377566000, "i" : 2014 }, "optimeDate" : ISODate("2013-01-16T23:06:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:07.732 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:09.733 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 m31002| Wed Jan 16 18:06:10.347 [rsMgr] replSet info two primaries (transiently) { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2614, "optime" : { "t" : 1358377570000, "i" : 3171 }, "optimeDate" : ISODate("2013-01-16T23:06:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2603, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2603, "optime" : { "t" : 1358377566000, "i" : 2244 }, "optimeDate" : ISODate("2013-01-16T23:06:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2613, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2601, "optime" : { "t" : 1358377570000, "i" : 2838 }, "optimeDate" : ISODate("2013-01-16T23:06:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:10Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2600, "optime" : { "t" : 1358377566000, "i" : 2244 }, "optimeDate" : ISODate("2013-01-16T23:06:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2600, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2612, "optime" : { "t" : 1358377570000, "i" : 2877 }, "optimeDate" : ISODate("2013-01-16T23:06:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:11.733 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:13.733 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2618, "optime" : { "t" : 1358377574000, "i" : 4306 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2607, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2607, "optime" : { "t" : 1358377574000, "i" : 227 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2617, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2605, "optime" : { "t" : 1358377574000, "i" : 2865 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2604, "optime" : { "t" : 1358377570000, "i" : 3171 }, "optimeDate" : ISODate("2013-01-16T23:06:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2604, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:14Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2616, "optime" : { "t" : 1358377574000, "i" : 3700 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:06:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Wed Jan 16 18:06:15.348 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:06:15.734 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:17.734 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:18.320 [conn159] end connection 127.0.0.3:59792 (3 connections now open) m31002| Wed Jan 16 18:06:18.320 [initandlisten] connection accepted from 127.0.0.3:46205 #160 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2622, "optime" : { "t" : 1358377578000, "i" : 6145 }, "optimeDate" : ISODate("2013-01-16T23:06:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2611, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2611, "optime" : { "t" : 1358377574000, "i" : 4306 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:16Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2621, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2609, "optime" : { "t" : 1358377578000, "i" : 2647 }, "optimeDate" : ISODate("2013-01-16T23:06:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2608, "optime" : { "t" : 1358377574000, "i" : 4306 }, "optimeDate" : ISODate("2013-01-16T23:06:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2608, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2620, "optime" : { "t" : 1358377578000, "i" : 6145 }, "optimeDate" : ISODate("2013-01-16T23:06:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:19.734 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:20.349 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:06:21.600 [conn158] end connection 127.0.0.2:54009 (3 connections now open) m31000| Wed Jan 16 18:06:21.735 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:22.100 [initandlisten] connection accepted from 127.0.0.2:48810 #161 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2626, "optime" : { "t" : 1358377582000, "i" : 7087 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2615, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2615, "optime" : { "t" : 1358377582000, "i" : 62 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2625, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2613, "optime" : { "t" : 1358377582000, "i" : 2768 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2612, "optime" : { "t" : 1358377578000, "i" : 6145 }, "optimeDate" : ISODate("2013-01-16T23:06:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:22Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2612, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:22Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2624, "optime" : { "t" : 1358377582000, "i" : 6899 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:23.735 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:06:24.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 18:06:25.349 [conn157] end connection 127.0.0.4:56803 (2 connections now open) m31000| Wed Jan 16 18:06:25.736 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:06:25.849 [initandlisten] connection accepted from 127.0.0.4:57349 #159 (3 connections now open) m31002| Wed Jan 16 18:06:25.850 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2630, "optime" : { "t" : 1358377586000, "i" : 8286 }, "optimeDate" : ISODate("2013-01-16T23:06:26Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2619, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2619, "optime" : { "t" : 1358377582000, "i" : 7087 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:24Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:26Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2629, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2617, "optime" : { "t" : 1358377586000, "i" : 2980 }, "optimeDate" : ISODate("2013-01-16T23:06:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:26Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2616, "optime" : { "t" : 1358377582000, "i" : 7087 }, "optimeDate" : ISODate("2013-01-16T23:06:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2616, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2628, "optime" : { "t" : 1358377586000, "i" : 7798 }, "optimeDate" : ISODate("2013-01-16T23:06:26Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:27.736 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:29.736 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:30.850 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2634, "optime" : { "t" : 1358377590000, "i" : 8616 }, "optimeDate" : ISODate("2013-01-16T23:06:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2623, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2623, "optime" : { "t" : 1358377586000, "i" : 8286 }, "optimeDate" : ISODate("2013-01-16T23:06:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:30Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2633, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2621, "optime" : { "t" : 1358377590000, "i" : 2817 }, "optimeDate" : ISODate("2013-01-16T23:06:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:30Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2620, "optime" : { "t" : 1358377590000, "i" : 8375 }, "optimeDate" : ISODate("2013-01-16T23:06:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2620, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2632, "optime" : { "t" : 1358377590000, "i" : 8201 }, "optimeDate" : ISODate("2013-01-16T23:06:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:06:31.737 [conn158] end connection 127.0.0.2:38974 (2 connections now open) m31001| Wed Jan 16 18:06:31.737 [initandlisten] connection accepted from 127.0.0.2:40951 #160 (3 connections now open) m31000| Wed Jan 16 18:06:31.737 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:33.737 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:34.579 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:310477 155ms m31001| Wed Jan 16 18:06:34.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:06:34.955 [conn98] end connection 127.0.0.4:35774 (4 connections now open) m31000| Wed Jan 16 18:06:34.955 [initandlisten] connection accepted from 127.0.0.4:43451 #99 (6 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2639, "optime" : { "t" : 1358377595000, "i" : 764 }, "optimeDate" : ISODate("2013-01-16T23:06:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2628, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2628, "optime" : { "t" : 1358377594000, "i" : 3895 }, "optimeDate" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2638, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2626, "optime" : { "t" : 1358377594000, "i" : 2227 }, "optimeDate" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2625, "optime" : { "t" : 1358377594000, "i" : 8398 }, "optimeDate" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2625, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:34Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2637, "optime" : { "t" : 1358377595000, "i" : 657 }, "optimeDate" : ISODate("2013-01-16T23:06:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:35.738 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:35.851 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:06:37.738 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:38.270 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:272015 136ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2643, "optime" : { "t" : 1358377599000, "i" : 1319 }, "optimeDate" : ISODate("2013-01-16T23:06:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2632, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2632, "optime" : { "t" : 1358377595000, "i" : 764 }, "optimeDate" : ISODate("2013-01-16T23:06:35Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2642, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2630, "optime" : { "t" : 1358377598000, "i" : 1778 }, "optimeDate" : ISODate("2013-01-16T23:06:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:38Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2629, "optime" : { "t" : 1358377598000, "i" : 8016 }, "optimeDate" : ISODate("2013-01-16T23:06:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2629, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2641, "optime" : { "t" : 1358377599000, "i" : 816 }, "optimeDate" : ISODate("2013-01-16T23:06:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:39.739 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:40.852 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:06:41.739 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2647, "optime" : { "t" : 1358377603000, "i" : 2628 }, "optimeDate" : ISODate("2013-01-16T23:06:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2636, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2636, "optime" : { "t" : 1358377599000, "i" : 1319 }, "optimeDate" : ISODate("2013-01-16T23:06:39Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2646, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2634, "optime" : { "t" : 1358377602000, "i" : 568 }, "optimeDate" : ISODate("2013-01-16T23:06:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2633, "optime" : { "t" : 1358377602000, "i" : 6500 }, "optimeDate" : ISODate("2013-01-16T23:06:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2633, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:42Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2645, "optime" : { "t" : 1358377603000, "i" : 2446 }, "optimeDate" : ISODate("2013-01-16T23:06:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:43.740 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:45.740 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:45.852 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2651, "optime" : { "t" : 1358377607000, "i" : 4179 }, "optimeDate" : ISODate("2013-01-16T23:06:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2640, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2640, "optime" : { "t" : 1358377607000, "i" : 970 }, "optimeDate" : ISODate("2013-01-16T23:06:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2650, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2638, "optime" : { "t" : 1358377606000, "i" : 3 }, "optimeDate" : ISODate("2013-01-16T23:06:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2637, "optime" : { "t" : 1358377606000, "i" : 5570 }, "optimeDate" : ISODate("2013-01-16T23:06:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:47Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2637, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:46Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2649, "optime" : { "t" : 1358377607000, "i" : 4179 }, "optimeDate" : ISODate("2013-01-16T23:06:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:47.740 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:48.325 [conn160] end connection 127.0.0.3:46205 (3 connections now open) m31002| Wed Jan 16 18:06:48.325 [initandlisten] connection accepted from 127.0.0.3:33870 #162 (4 connections now open) m31000| Wed Jan 16 18:06:49.741 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:50.853 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2655, "optime" : { "t" : 1358377611000, "i" : 5961 }, "optimeDate" : ISODate("2013-01-16T23:06:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2644, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2644, "optime" : { "t" : 1358377607000, "i" : 4179 }, "optimeDate" : ISODate("2013-01-16T23:06:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:50Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2654, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2642, "optime" : { "t" : 1358377607000, "i" : 4179 }, "optimeDate" : ISODate("2013-01-16T23:06:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:50Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2641, "optime" : { "t" : 1358377610000, "i" : 3719 }, "optimeDate" : ISODate("2013-01-16T23:06:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2641, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2653, "optime" : { "t" : 1358377611000, "i" : 5242 }, "optimeDate" : ISODate("2013-01-16T23:06:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:06:51.741 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:53.741 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:06:55.742 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2659, "optime" : { "t" : 1358377615000, "i" : 6710 }, "optimeDate" : ISODate("2013-01-16T23:06:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2648, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2648, "optime" : { "t" : 1358377611000, "i" : 5961 }, "optimeDate" : ISODate("2013-01-16T23:06:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2658, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2646, "optime" : { "t" : 1358377611000, "i" : 5961 }, "optimeDate" : ISODate("2013-01-16T23:06:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2645, "optime" : { "t" : 1358377614000, "i" : 2816 }, "optimeDate" : ISODate("2013-01-16T23:06:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2645, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:54Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2657, "optime" : { "t" : 1358377615000, "i" : 6710 }, "optimeDate" : ISODate("2013-01-16T23:06:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:06:55.854 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:06:56.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:06:57.742 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:06:59.605 [conn161] end connection 127.0.0.2:48810 (3 connections now open) m31000| Wed Jan 16 18:06:59.742 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2663, "optime" : { "t" : 1358377619000, "i" : 7848 }, "optimeDate" : ISODate("2013-01-16T23:06:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2652, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2652, "optime" : { "t" : 1358377615000, "i" : 6710 }, "optimeDate" : ISODate("2013-01-16T23:06:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:58Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2662, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2650, "optime" : { "t" : 1358377615000, "i" : 6710 }, "optimeDate" : ISODate("2013-01-16T23:06:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:06:58Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:06:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2649, "optime" : { "t" : 1358377618000, "i" : 1313 }, "optimeDate" : ISODate("2013-01-16T23:06:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2649, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2661, "optime" : { "t" : 1358377619000, "i" : 7536 }, "optimeDate" : ISODate("2013-01-16T23:06:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:07:00.105 [initandlisten] connection accepted from 127.0.0.2:49940 #163 (4 connections now open) m31002| Wed Jan 16 18:07:00.854 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:07:01.743 [conn160] end connection 127.0.0.2:40951 (2 connections now open) m31001| Wed Jan 16 18:07:01.743 [initandlisten] connection accepted from 127.0.0.2:46686 #161 (3 connections now open) m31000| Wed Jan 16 18:07:01.743 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:07:03.354 [conn159] end connection 127.0.0.4:57349 (2 connections now open) m31000| Wed Jan 16 18:07:03.743 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:07:03.855 [initandlisten] connection accepted from 127.0.0.4:38197 #162 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2667, "optime" : { "t" : 1358377623000, "i" : 8367 }, "optimeDate" : ISODate("2013-01-16T23:07:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2656, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2656, "optime" : { "t" : 1358377619000, "i" : 7848 }, "optimeDate" : ISODate("2013-01-16T23:06:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:02Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2666, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2654, "optime" : { "t" : 1358377619000, "i" : 7848 }, "optimeDate" : ISODate("2013-01-16T23:06:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:03Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2653, "optime" : { "t" : 1358377622000, "i" : 1406 }, "optimeDate" : ISODate("2013-01-16T23:07:02Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2653, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2665, "optime" : { "t" : 1358377623000, "i" : 8287 }, "optimeDate" : ISODate("2013-01-16T23:07:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:04.960 [conn99] end connection 127.0.0.4:43451 (4 connections now open) m31000| Wed Jan 16 18:07:04.960 [initandlisten] connection accepted from 127.0.0.4:60352 #100 (5 connections now open) m31000| Wed Jan 16 18:07:05.744 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:06.355 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:07:06.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:07:07.744 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2672, "optime" : { "t" : 1358377628000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T23:07:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2661, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2661, "optime" : { "t" : 1358377627000, "i" : 5225 }, "optimeDate" : ISODate("2013-01-16T23:07:07Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2671, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2659, "optime" : { "t" : 1358377623000, "i" : 8367 }, "optimeDate" : ISODate("2013-01-16T23:07:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:06Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2658, "optime" : { "t" : 1358377623000, "i" : 8367 }, "optimeDate" : ISODate("2013-01-16T23:07:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2658, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2670, "optime" : { "t" : 1358377628000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T23:07:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:09.745 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:11.356 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:11.745 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2676, "optime" : { "t" : 1358377632000, "i" : 3142 }, "optimeDate" : ISODate("2013-01-16T23:07:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2665, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2665, "optime" : { "t" : 1358377628000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T23:07:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:10Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2675, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2663, "optime" : { "t" : 1358377628000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T23:07:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2662, "optime" : { "t" : 1358377628000, "i" : 1108 }, "optimeDate" : ISODate("2013-01-16T23:07:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2662, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2674, "optime" : { "t" : 1358377632000, "i" : 2826 }, "optimeDate" : ISODate("2013-01-16T23:07:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:13.745 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:15.745 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:16.356 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2680, "optime" : { "t" : 1358377636000, "i" : 3941 }, "optimeDate" : ISODate("2013-01-16T23:07:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2669, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2669, "optime" : { "t" : 1358377632000, "i" : 3142 }, "optimeDate" : ISODate("2013-01-16T23:07:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:06:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2679, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2667, "optime" : { "t" : 1358377636000, "i" : 2768 }, "optimeDate" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:16Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2666, "optime" : { "t" : 1358377632000, "i" : 3142 }, "optimeDate" : ISODate("2013-01-16T23:07:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2666, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2678, "optime" : { "t" : 1358377636000, "i" : 3941 }, "optimeDate" : ISODate("2013-01-16T23:07:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:17.746 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:18.331 [conn162] end connection 127.0.0.3:33870 (3 connections now open) m31002| Wed Jan 16 18:07:18.331 [initandlisten] connection accepted from 127.0.0.3:42216 #164 (4 connections now open) m31000| Wed Jan 16 18:07:19.746 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2684, "optime" : { "t" : 1358377640000, "i" : 5229 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2673, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2673, "optime" : { "t" : 1358377640000, "i" : 609 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2683, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2671, "optime" : { "t" : 1358377640000, "i" : 2840 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2670, "optime" : { "t" : 1358377636000, "i" : 3941 }, "optimeDate" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2670, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:20Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2682, "optime" : { "t" : 1358377640000, "i" : 5229 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:07:21.357 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:21.746 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:23.747 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:24.473 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:253459 126ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2688, "optime" : { "t" : 1358377644000, "i" : 6570 }, "optimeDate" : ISODate("2013-01-16T23:07:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2677, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2677, "optime" : { "t" : 1358377640000, "i" : 5229 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:22Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2687, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2675, "optime" : { "t" : 1358377644000, "i" : 2964 }, "optimeDate" : ISODate("2013-01-16T23:07:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2674, "optime" : { "t" : 1358377640000, "i" : 5229 }, "optimeDate" : ISODate("2013-01-16T23:07:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2674, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2686, "optime" : { "t" : 1358377644000, "i" : 6570 }, "optimeDate" : ISODate("2013-01-16T23:07:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:25.747 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:26.358 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:27.747 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2692, "optime" : { "t" : 1358377648000, "i" : 7788 }, "optimeDate" : ISODate("2013-01-16T23:07:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2681, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2681, "optime" : { "t" : 1358377644000, "i" : 6570 }, "optimeDate" : ISODate("2013-01-16T23:07:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2691, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2679, "optime" : { "t" : 1358377648000, "i" : 2893 }, "optimeDate" : ISODate("2013-01-16T23:07:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2678, "optime" : { "t" : 1358377644000, "i" : 6570 }, "optimeDate" : ISODate("2013-01-16T23:07:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2678, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2690, "optime" : { "t" : 1358377648000, "i" : 7788 }, "optimeDate" : ISODate("2013-01-16T23:07:28Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:07:28.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:07:29.748 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:31.358 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:07:31.748 [conn161] end connection 127.0.0.2:46686 (2 connections now open) m31001| Wed Jan 16 18:07:31.748 [initandlisten] connection accepted from 127.0.0.2:44797 #163 (3 connections now open) m31000| Wed Jan 16 18:07:31.748 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2696, "optime" : { "t" : 1358377652000, "i" : 8579 }, "optimeDate" : ISODate("2013-01-16T23:07:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2685, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2685, "optime" : { "t" : 1358377652000, "i" : 5272 }, "optimeDate" : ISODate("2013-01-16T23:07:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2695, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2683, "optime" : { "t" : 1358377652000, "i" : 2939 }, "optimeDate" : ISODate("2013-01-16T23:07:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2682, "optime" : { "t" : 1358377648000, "i" : 7788 }, "optimeDate" : ISODate("2013-01-16T23:07:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2682, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:32Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2694, "optime" : { "t" : 1358377652000, "i" : 8579 }, "optimeDate" : ISODate("2013-01-16T23:07:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:33.749 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:34.966 [conn100] end connection 127.0.0.4:60352 (4 connections now open) m31000| Wed Jan 16 18:07:34.966 [initandlisten] connection accepted from 127.0.0.4:48808 #101 (6 connections now open) m31000| Wed Jan 16 18:07:35.749 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:36.359 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:36.943 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:359182 179ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2701, "optime" : { "t" : 1358377657000, "i" : 1523 }, "optimeDate" : ISODate("2013-01-16T23:07:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2690, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2690, "optime" : { "t" : 1358377652000, "i" : 8579 }, "optimeDate" : ISODate("2013-01-16T23:07:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:36Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2700, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2688, "optime" : { "t" : 1358377656000, "i" : 2809 }, "optimeDate" : ISODate("2013-01-16T23:07:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:36Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2687, "optime" : { "t" : 1358377656000, "i" : 7419 }, "optimeDate" : ISODate("2013-01-16T23:07:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2687, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2699, "optime" : { "t" : 1358377657000, "i" : 1141 }, "optimeDate" : ISODate("2013-01-16T23:07:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:07:37.610 [conn163] end connection 127.0.0.2:49940 (3 connections now open) m31000| Wed Jan 16 18:07:37.749 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:38.110 [initandlisten] connection accepted from 127.0.0.2:58668 #165 (4 connections now open) m31001| Wed Jan 16 18:07:38.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:07:39.750 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2705, "optime" : { "t" : 1358377661000, "i" : 2731 }, "optimeDate" : ISODate("2013-01-16T23:07:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2694, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2694, "optime" : { "t" : 1358377660000, "i" : 4395 }, "optimeDate" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2704, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2692, "optime" : { "t" : 1358377660000, "i" : 1592 }, "optimeDate" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2691, "optime" : { "t" : 1358377660000, "i" : 7259 }, "optimeDate" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2691, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:40Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2703, "optime" : { "t" : 1358377661000, "i" : 2731 }, "optimeDate" : ISODate("2013-01-16T23:07:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:07:41.360 [conn162] end connection 127.0.0.4:38197 (2 connections now open) m31000| Wed Jan 16 18:07:41.750 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:07:41.860 [initandlisten] connection accepted from 127.0.0.4:40167 #164 (3 connections now open) m31002| Wed Jan 16 18:07:41.860 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:43.751 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2709, "optime" : { "t" : 1358377665000, "i" : 3369 }, "optimeDate" : ISODate("2013-01-16T23:07:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2698, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2698, "optime" : { "t" : 1358377661000, "i" : 2731 }, "optimeDate" : ISODate("2013-01-16T23:07:41Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2708, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2696, "optime" : { "t" : 1358377664000, "i" : 792 }, "optimeDate" : ISODate("2013-01-16T23:07:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:44Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2695, "optime" : { "t" : 1358377664000, "i" : 6381 }, "optimeDate" : ISODate("2013-01-16T23:07:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2695, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2707, "optime" : { "t" : 1358377665000, "i" : 3369 }, "optimeDate" : ISODate("2013-01-16T23:07:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:45.751 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:46.861 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:47.751 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:48.335 [conn164] end connection 127.0.0.3:42216 (3 connections now open) m31002| Wed Jan 16 18:07:48.336 [initandlisten] connection accepted from 127.0.0.3:44876 #166 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2713, "optime" : { "t" : 1358377669000, "i" : 4541 }, "optimeDate" : ISODate("2013-01-16T23:07:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2702, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2702, "optime" : { "t" : 1358377665000, "i" : 3369 }, "optimeDate" : ISODate("2013-01-16T23:07:45Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:48Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2712, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2700, "optime" : { "t" : 1358377665000, "i" : 3369 }, "optimeDate" : ISODate("2013-01-16T23:07:45Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:49Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2699, "optime" : { "t" : 1358377668000, "i" : 4482 }, "optimeDate" : ISODate("2013-01-16T23:07:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2699, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2711, "optime" : { "t" : 1358377669000, "i" : 4541 }, "optimeDate" : ISODate("2013-01-16T23:07:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:49.752 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:51.752 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:51.861 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2717, "optime" : { "t" : 1358377673000, "i" : 5883 }, "optimeDate" : ISODate("2013-01-16T23:07:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2706, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2706, "optime" : { "t" : 1358377673000, "i" : 787 }, "optimeDate" : ISODate("2013-01-16T23:07:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2716, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2704, "optime" : { "t" : 1358377669000, "i" : 4541 }, "optimeDate" : ISODate("2013-01-16T23:07:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2703, "optime" : { "t" : 1358377672000, "i" : 3458 }, "optimeDate" : ISODate("2013-01-16T23:07:52Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2703, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:52Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2715, "optime" : { "t" : 1358377673000, "i" : 5122 }, "optimeDate" : ISODate("2013-01-16T23:07:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:53.753 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:07:55.753 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:07:56.862 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:07:57.753 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2721, "optime" : { "t" : 1358377677000, "i" : 7008 }, "optimeDate" : ISODate("2013-01-16T23:07:57Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2710, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2710, "optime" : { "t" : 1358377673000, "i" : 5883 }, "optimeDate" : ISODate("2013-01-16T23:07:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:56Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:57Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:57Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2720, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2708, "optime" : { "t" : 1358377673000, "i" : 5883 }, "optimeDate" : ISODate("2013-01-16T23:07:53Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:07:56Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:07:57Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2707, "optime" : { "t" : 1358377676000, "i" : 2916 }, "optimeDate" : ISODate("2013-01-16T23:07:56Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2707, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2719, "optime" : { "t" : 1358377677000, "i" : 7008 }, "optimeDate" : ISODate("2013-01-16T23:07:57Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:07:59.754 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:08:00.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Wed Jan 16 18:08:01.754 [conn163] end connection 127.0.0.2:44797 (2 connections now open) m31001| Wed Jan 16 18:08:01.754 [initandlisten] connection accepted from 127.0.0.2:42240 #165 (3 connections now open) m31000| Wed Jan 16 18:08:01.754 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:01.863 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2725, "optime" : { "t" : 1358377681000, "i" : 8635 }, "optimeDate" : ISODate("2013-01-16T23:08:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2714, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2714, "optime" : { "t" : 1358377677000, "i" : 7008 }, "optimeDate" : ISODate("2013-01-16T23:07:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:01Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2724, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2712, "optime" : { "t" : 1358377677000, "i" : 7008 }, "optimeDate" : ISODate("2013-01-16T23:07:57Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:01Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2711, "optime" : { "t" : 1358377680000, "i" : 1061 }, "optimeDate" : ISODate("2013-01-16T23:08:00Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2711, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2723, "optime" : { "t" : 1358377681000, "i" : 8541 }, "optimeDate" : ISODate("2013-01-16T23:08:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:03.755 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:04.971 [conn101] end connection 127.0.0.4:48808 (4 connections now open) m31000| Wed Jan 16 18:08:04.971 [initandlisten] connection accepted from 127.0.0.4:56144 #102 (5 connections now open) m31000| Wed Jan 16 18:08:05.429 [journal] old journal file will be removed: /data/db/testReplSet-0/journal/j._3 m31000| Wed Jan 16 18:08:05.505 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:364275 182ms m31000| Wed Jan 16 18:08:05.755 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2730, "optime" : { "t" : 1358377686000, "i" : 228 }, "optimeDate" : ISODate("2013-01-16T23:08:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2719, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2719, "optime" : { "t" : 1358377685000, "i" : 4944 }, "optimeDate" : ISODate("2013-01-16T23:08:05Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2729, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2717, "optime" : { "t" : 1358377681000, "i" : 8635 }, "optimeDate" : ISODate("2013-01-16T23:08:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:04Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2716, "optime" : { "t" : 1358377681000, "i" : 8635 }, "optimeDate" : ISODate("2013-01-16T23:08:01Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2716, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2728, "optime" : { "t" : 1358377685000, "i" : 9666 }, "optimeDate" : ISODate("2013-01-16T23:08:05Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:08:06.863 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:08:07.755 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:09.589 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:276867 138ms m31000| Wed Jan 16 18:08:09.755 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2734, "optime" : { "t" : 1358377690000, "i" : 1694 }, "optimeDate" : ISODate("2013-01-16T23:08:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2723, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2723, "optime" : { "t" : 1358377686000, "i" : 228 }, "optimeDate" : ISODate("2013-01-16T23:08:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:08Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2733, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2721, "optime" : { "t" : 1358377686000, "i" : 228 }, "optimeDate" : ISODate("2013-01-16T23:08:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2720, "optime" : { "t" : 1358377686000, "i" : 228 }, "optimeDate" : ISODate("2013-01-16T23:08:06Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2720, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2732, "optime" : { "t" : 1358377690000, "i" : 1694 }, "optimeDate" : ISODate("2013-01-16T23:08:10Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:08:10.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:08:11.756 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:11.864 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:08:13.756 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2738, "optime" : { "t" : 1358377694000, "i" : 3785 }, "optimeDate" : ISODate("2013-01-16T23:08:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2727, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2727, "optime" : { "t" : 1358377690000, "i" : 1694 }, "optimeDate" : ISODate("2013-01-16T23:08:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2737, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2725, "optime" : { "t" : 1358377694000, "i" : 2629 }, "optimeDate" : ISODate("2013-01-16T23:08:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:14Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2724, "optime" : { "t" : 1358377690000, "i" : 1694 }, "optimeDate" : ISODate("2013-01-16T23:08:10Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2724, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2736, "optime" : { "t" : 1358377694000, "i" : 3638 }, "optimeDate" : ISODate("2013-01-16T23:08:14Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:08:15.615 [conn165] end connection 127.0.0.2:58668 (3 connections now open) m31000| Wed Jan 16 18:08:15.756 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:16.115 [initandlisten] connection accepted from 127.0.0.2:41033 #167 (4 connections now open) m31002| Wed Jan 16 18:08:16.865 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:08:17.757 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:18.341 [conn166] end connection 127.0.0.3:44876 (3 connections now open) m31002| Wed Jan 16 18:08:18.341 [initandlisten] connection accepted from 127.0.0.3:58456 #168 (4 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2742, "optime" : { "t" : 1358377698000, "i" : 6440 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2731, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2731, "optime" : { "t" : 1358377698000, "i" : 4531 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:18Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:07:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:17Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2741, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2729, "optime" : { "t" : 1358377698000, "i" : 2772 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:18Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2728, "optime" : { "t" : 1358377694000, "i" : 3785 }, "optimeDate" : ISODate("2013-01-16T23:08:14Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2728, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:18Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2740, "optime" : { "t" : 1358377698000, "i" : 6440 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:08:19.365 [conn164] end connection 127.0.0.4:40167 (2 connections now open) m31000| Wed Jan 16 18:08:19.757 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:08:19.865 [initandlisten] connection accepted from 127.0.0.4:32995 #166 (3 connections now open) m31000| Wed Jan 16 18:08:21.758 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:22.365 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2746, "optime" : { "t" : 1358377702000, "i" : 8665 }, "optimeDate" : ISODate("2013-01-16T23:08:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2735, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2735, "optime" : { "t" : 1358377698000, "i" : 6440 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2745, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2733, "optime" : { "t" : 1358377702000, "i" : 3129 }, "optimeDate" : ISODate("2013-01-16T23:08:22Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:22Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2732, "optime" : { "t" : 1358377698000, "i" : 6440 }, "optimeDate" : ISODate("2013-01-16T23:08:18Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:21Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2732, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2744, "optime" : { "t" : 1358377702000, "i" : 8633 }, "optimeDate" : ISODate("2013-01-16T23:08:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:23.758 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:25.758 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2751, "optime" : { "t" : 1358377707000, "i" : 462 }, "optimeDate" : ISODate("2013-01-16T23:08:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2740, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2740, "optime" : { "t" : 1358377706000, "i" : 1100 }, "optimeDate" : ISODate("2013-01-16T23:08:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:26Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2750, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2738, "optime" : { "t" : 1358377706000, "i" : 2975 }, "optimeDate" : ISODate("2013-01-16T23:08:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2737, "optime" : { "t" : 1358377706000, "i" : 8810 }, "optimeDate" : ISODate("2013-01-16T23:08:26Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2737, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:26Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2749, "optime" : { "t" : 1358377707000, "i" : 454 }, "optimeDate" : ISODate("2013-01-16T23:08:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:08:27.366 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:08:27.759 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:29.759 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2755, "optime" : { "t" : 1358377711000, "i" : 2202 }, "optimeDate" : ISODate("2013-01-16T23:08:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2744, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2744, "optime" : { "t" : 1358377711000, "i" : 789 }, "optimeDate" : ISODate("2013-01-16T23:08:31Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2754, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2742, "optime" : { "t" : 1358377710000, "i" : 1634 }, "optimeDate" : ISODate("2013-01-16T23:08:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2741, "optime" : { "t" : 1358377710000, "i" : 7422 }, "optimeDate" : ISODate("2013-01-16T23:08:30Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2741, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:30Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2753, "optime" : { "t" : 1358377711000, "i" : 2202 }, "optimeDate" : ISODate("2013-01-16T23:08:31Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:08:31.760 [conn165] end connection 127.0.0.2:42240 (2 connections now open) m31001| Wed Jan 16 18:08:31.760 [initandlisten] connection accepted from 127.0.0.2:60461 #167 (3 connections now open) m31000| Wed Jan 16 18:08:31.760 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:32.367 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:08:32.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:08:33.760 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:34.470 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:304314 152ms m31000| Wed Jan 16 18:08:34.977 [conn102] end connection 127.0.0.4:56144 (4 connections now open) m31000| Wed Jan 16 18:08:34.977 [initandlisten] connection accepted from 127.0.0.4:47614 #103 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2759, "optime" : { "t" : 1358377715000, "i" : 2211 }, "optimeDate" : ISODate("2013-01-16T23:08:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2748, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2748, "optime" : { "t" : 1358377711000, "i" : 2202 }, "optimeDate" : ISODate("2013-01-16T23:08:31Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:34Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2758, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2746, "optime" : { "t" : 1358377714000, "i" : 740 }, "optimeDate" : ISODate("2013-01-16T23:08:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:34Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2745, "optime" : { "t" : 1358377714000, "i" : 7625 }, "optimeDate" : ISODate("2013-01-16T23:08:34Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2745, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2757, "optime" : { "t" : 1358377715000, "i" : 1837 }, "optimeDate" : ISODate("2013-01-16T23:08:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:35.761 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:37.368 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:08:37.761 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2763, "optime" : { "t" : 1358377719000, "i" : 3606 }, "optimeDate" : ISODate("2013-01-16T23:08:39Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2752, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2752, "optime" : { "t" : 1358377718000, "i" : 3062 }, "optimeDate" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:38Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:39Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2762, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2750, "optime" : { "t" : 1358377718000, "i" : 183 }, "optimeDate" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:39Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2749, "optime" : { "t" : 1358377718000, "i" : 6183 }, "optimeDate" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2749, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:38Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2761, "optime" : { "t" : 1358377719000, "i" : 3606 }, "optimeDate" : ISODate("2013-01-16T23:08:39Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:39.762 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:41.762 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:42.368 [rsMgr] replSet info two primaries (transiently) m31001| Wed Jan 16 18:08:42.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2767, "optime" : { "t" : 1358377723000, "i" : 4635 }, "optimeDate" : ISODate("2013-01-16T23:08:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2756, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2756, "optime" : { "t" : 1358377719000, "i" : 3606 }, "optimeDate" : ISODate("2013-01-16T23:08:39Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:42Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:41Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2766, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2754, "optime" : { "t" : 1358377719000, "i" : 3606 }, "optimeDate" : ISODate("2013-01-16T23:08:39Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:42Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2753, "optime" : { "t" : 1358377722000, "i" : 5208 }, "optimeDate" : ISODate("2013-01-16T23:08:42Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2753, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2765, "optime" : { "t" : 1358377723000, "i" : 4635 }, "optimeDate" : ISODate("2013-01-16T23:08:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:43.762 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:45.763 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:47.369 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2771, "optime" : { "t" : 1358377727000, "i" : 5862 }, "optimeDate" : ISODate("2013-01-16T23:08:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2760, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2760, "optime" : { "t" : 1358377723000, "i" : 4635 }, "optimeDate" : ISODate("2013-01-16T23:08:43Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:46Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:45Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2770, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2758, "optime" : { "t" : 1358377723000, "i" : 4635 }, "optimeDate" : ISODate("2013-01-16T23:08:43Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2757, "optime" : { "t" : 1358377726000, "i" : 3978 }, "optimeDate" : ISODate("2013-01-16T23:08:46Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2757, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2769, "optime" : { "t" : 1358377727000, "i" : 5862 }, "optimeDate" : ISODate("2013-01-16T23:08:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:08:47.763 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:48.347 [conn168] end connection 127.0.0.3:58456 (3 connections now open) m31002| Wed Jan 16 18:08:48.347 [initandlisten] connection accepted from 127.0.0.3:33365 #169 (4 connections now open) m31000| Wed Jan 16 18:08:49.764 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:08:51.764 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2775, "optime" : { "t" : 1358377731000, "i" : 7311 }, "optimeDate" : ISODate("2013-01-16T23:08:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2764, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2764, "optime" : { "t" : 1358377731000, "i" : 480 }, "optimeDate" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2774, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2762, "optime" : { "t" : 1358377727000, "i" : 5862 }, "optimeDate" : ISODate("2013-01-16T23:08:47Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2761, "optime" : { "t" : 1358377730000, "i" : 2383 }, "optimeDate" : ISODate("2013-01-16T23:08:50Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:51Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2761, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:50Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2773, "optime" : { "t" : 1358377731000, "i" : 7311 }, "optimeDate" : ISODate("2013-01-16T23:08:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:08:52.370 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:08:53.620 [conn167] end connection 127.0.0.2:41033 (3 connections now open) m31000| Wed Jan 16 18:08:53.764 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:08:54.121 [initandlisten] connection accepted from 127.0.0.2:53049 #170 (4 connections now open) m31000| Wed Jan 16 18:08:55.765 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2779, "optime" : { "t" : 1358377735000, "i" : 8534 }, "optimeDate" : ISODate("2013-01-16T23:08:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2768, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2768, "optime" : { "t" : 1358377731000, "i" : 7311 }, "optimeDate" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:54Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2778, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2766, "optime" : { "t" : 1358377731000, "i" : 7311 }, "optimeDate" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:54Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2765, "optime" : { "t" : 1358377734000, "i" : 1276 }, "optimeDate" : ISODate("2013-01-16T23:08:54Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2765, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2777, "optime" : { "t" : 1358377735000, "i" : 8420 }, "optimeDate" : ISODate("2013-01-16T23:08:55Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:08:57.370 [conn166] end connection 127.0.0.4:32995 (2 connections now open) m31000| Wed Jan 16 18:08:57.765 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:08:57.870 [initandlisten] connection accepted from 127.0.0.4:55559 #168 (3 connections now open) m31002| Wed Jan 16 18:08:57.871 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:08:59.001 [journal] old journal file will be removed: /data/db/testReplSet-2/journal/j._3 m31000| Wed Jan 16 18:08:59.765 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2783, "optime" : { "t" : 1358377739000, "i" : 9169 }, "optimeDate" : ISODate("2013-01-16T23:08:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2772, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2772, "optime" : { "t" : 1358377738000, "i" : 260 }, "optimeDate" : ISODate("2013-01-16T23:08:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2782, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2770, "optime" : { "t" : 1358377735000, "i" : 8534 }, "optimeDate" : ISODate("2013-01-16T23:08:55Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:08:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2769, "optime" : { "t" : 1358377738000, "i" : 626 }, "optimeDate" : ISODate("2013-01-16T23:08:58Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2769, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:08:58Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2781, "optime" : { "t" : 1358377739000, "i" : 9169 }, "optimeDate" : ISODate("2013-01-16T23:08:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:09:01.766 [conn167] end connection 127.0.0.2:60461 (2 connections now open) m31001| Wed Jan 16 18:09:01.766 [initandlisten] connection accepted from 127.0.0.2:37325 #169 (3 connections now open) m31000| Wed Jan 16 18:09:01.766 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:02.871 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:03.766 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2788, "optime" : { "t" : 1358377744000, "i" : 1385 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2777, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2777, "optime" : { "t" : 1358377744000, "i" : 1020 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2787, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2775, "optime" : { "t" : 1358377739000, "i" : 9169 }, "optimeDate" : ISODate("2013-01-16T23:08:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:02Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2774, "optime" : { "t" : 1358377739000, "i" : 9169 }, "optimeDate" : ISODate("2013-01-16T23:08:59Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2774, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2786, "optime" : { "t" : 1358377744000, "i" : 1385 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31001| Wed Jan 16 18:09:04.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:09:04.982 [conn103] end connection 127.0.0.4:47614 (4 connections now open) m31000| Wed Jan 16 18:09:04.982 [initandlisten] connection accepted from 127.0.0.4:51437 #104 (5 connections now open) m31000| Wed Jan 16 18:09:05.767 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:09:07.767 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:07.872 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2792, "optime" : { "t" : 1358377748000, "i" : 2565 }, "optimeDate" : ISODate("2013-01-16T23:09:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2781, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2781, "optime" : { "t" : 1358377744000, "i" : 1385 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:06Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:07Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2791, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2779, "optime" : { "t" : 1358377744000, "i" : 1385 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:07Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2778, "optime" : { "t" : 1358377744000, "i" : 1385 }, "optimeDate" : ISODate("2013-01-16T23:09:04Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2778, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2790, "optime" : { "t" : 1358377748000, "i" : 2474 }, "optimeDate" : ISODate("2013-01-16T23:09:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:09.767 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:09:11.768 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2796, "optime" : { "t" : 1358377752000, "i" : 3744 }, "optimeDate" : ISODate("2013-01-16T23:09:12Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2785, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2785, "optime" : { "t" : 1358377751000, "i" : 2018 }, "optimeDate" : ISODate("2013-01-16T23:09:11Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:12Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2795, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2783, "optime" : { "t" : 1358377752000, "i" : 3034 }, "optimeDate" : ISODate("2013-01-16T23:09:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:12Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2782, "optime" : { "t" : 1358377748000, "i" : 2565 }, "optimeDate" : ISODate("2013-01-16T23:09:08Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:11Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2782, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:12Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2794, "optime" : { "t" : 1358377752000, "i" : 3744 }, "optimeDate" : ISODate("2013-01-16T23:09:12Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31002| Wed Jan 16 18:09:12.873 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:13.768 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:09:14.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:09:15.768 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2800, "optime" : { "t" : 1358377756000, "i" : 5271 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2789, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2789, "optime" : { "t" : 1358377756000, "i" : 5234 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2799, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2787, "optime" : { "t" : 1358377756000, "i" : 2943 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2786, "optime" : { "t" : 1358377752000, "i" : 3744 }, "optimeDate" : ISODate("2013-01-16T23:09:12Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2786, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:16Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2798, "optime" : { "t" : 1358377756000, "i" : 5271 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:17.769 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:17.873 [rsMgr] replSet info two primaries (transiently) m31002| Wed Jan 16 18:09:18.352 [conn169] end connection 127.0.0.3:33365 (3 connections now open) m31002| Wed Jan 16 18:09:18.352 [initandlisten] connection accepted from 127.0.0.3:36979 #171 (4 connections now open) m31000| Wed Jan 16 18:09:19.769 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2804, "optime" : { "t" : 1358377760000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T23:09:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2793, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2793, "optime" : { "t" : 1358377756000, "i" : 5271 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:08:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2803, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2791, "optime" : { "t" : 1358377760000, "i" : 2896 }, "optimeDate" : ISODate("2013-01-16T23:09:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:20Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2790, "optime" : { "t" : 1358377756000, "i" : 5271 }, "optimeDate" : ISODate("2013-01-16T23:09:16Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2790, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2802, "optime" : { "t" : 1358377760000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T23:09:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:21.769 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:22.874 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:23.770 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2808, "optime" : { "t" : 1358377764000, "i" : 8490 }, "optimeDate" : ISODate("2013-01-16T23:09:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2797, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2797, "optime" : { "t" : 1358377764000, "i" : 1295 }, "optimeDate" : ISODate("2013-01-16T23:09:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2807, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2795, "optime" : { "t" : 1358377764000, "i" : 3293 }, "optimeDate" : ISODate("2013-01-16T23:09:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2794, "optime" : { "t" : 1358377760000, "i" : 6929 }, "optimeDate" : ISODate("2013-01-16T23:09:20Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2794, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:24Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2806, "optime" : { "t" : 1358377764000, "i" : 8490 }, "optimeDate" : ISODate("2013-01-16T23:09:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:25.770 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:09:27.770 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:27.875 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2813, "optime" : { "t" : 1358377769000, "i" : 70 }, "optimeDate" : ISODate("2013-01-16T23:09:29Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2802, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2802, "optime" : { "t" : 1358377764000, "i" : 8490 }, "optimeDate" : ISODate("2013-01-16T23:09:24Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:28Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:29Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2812, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2800, "optime" : { "t" : 1358377768000, "i" : 3401 }, "optimeDate" : ISODate("2013-01-16T23:09:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:29Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2799, "optime" : { "t" : 1358377768000, "i" : 9113 }, "optimeDate" : ISODate("2013-01-16T23:09:28Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2799, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:28Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2811, "optime" : { "t" : 1358377769000, "i" : 70 }, "optimeDate" : ISODate("2013-01-16T23:09:29Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:29.771 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:31.626 [conn170] end connection 127.0.0.2:53049 (3 connections now open) m31001| Wed Jan 16 18:09:31.771 [conn169] end connection 127.0.0.2:37325 (2 connections now open) m31001| Wed Jan 16 18:09:31.771 [initandlisten] connection accepted from 127.0.0.2:41838 #170 (4 connections now open) m31000| Wed Jan 16 18:09:31.772 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:32.126 [initandlisten] connection accepted from 127.0.0.2:33378 #172 (4 connections now open) m31002| Wed Jan 16 18:09:32.875 [rsMgr] replSet info two primaries (transiently) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2817, "optime" : { "t" : 1358377773000, "i" : 1729 }, "optimeDate" : ISODate("2013-01-16T23:09:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2806, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2806, "optime" : { "t" : 1358377772000, "i" : 745 }, "optimeDate" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:32Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:31Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2816, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2804, "optime" : { "t" : 1358377772000, "i" : 2440 }, "optimeDate" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:32Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2803, "optime" : { "t" : 1358377772000, "i" : 8158 }, "optimeDate" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2803, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2815, "optime" : { "t" : 1358377773000, "i" : 1729 }, "optimeDate" : ISODate("2013-01-16T23:09:33Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:33.772 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Wed Jan 16 18:09:34.988 [conn104] end connection 127.0.0.4:51437 (4 connections now open) m31000| Wed Jan 16 18:09:34.988 [initandlisten] connection accepted from 127.0.0.4:58600 #105 (5 connections now open) m31001| Wed Jan 16 18:09:35.376 [conn168] end connection 127.0.0.4:55559 (2 connections now open) m31000| Wed Jan 16 18:09:35.772 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:09:35.876 [initandlisten] connection accepted from 127.0.0.4:42405 #171 (3 connections now open) m31001| Wed Jan 16 18:09:36.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:09:36.906 [conn1] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:243527 121ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2821, "optime" : { "t" : 1358377777000, "i" : 3251 }, "optimeDate" : ISODate("2013-01-16T23:09:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2810, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2810, "optime" : { "t" : 1358377777000, "i" : 1224 }, "optimeDate" : ISODate("2013-01-16T23:09:37Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2820, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2808, "optime" : { "t" : 1358377776000, "i" : 616 }, "optimeDate" : ISODate("2013-01-16T23:09:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2807, "optime" : { "t" : 1358377776000, "i" : 6505 }, "optimeDate" : ISODate("2013-01-16T23:09:36Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2807, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:36Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2819, "optime" : { "t" : 1358377777000, "i" : 3251 }, "optimeDate" : ISODate("2013-01-16T23:09:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:37.773 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:38.376 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:39.773 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2825, "optime" : { "t" : 1358377781000, "i" : 4497 }, "optimeDate" : ISODate("2013-01-16T23:09:41Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2814, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2814, "optime" : { "t" : 1358377777000, "i" : 3251 }, "optimeDate" : ISODate("2013-01-16T23:09:37Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:40Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:41Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:39Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2824, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2812, "optime" : { "t" : 1358377777000, "i" : 3251 }, "optimeDate" : ISODate("2013-01-16T23:09:37Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:40Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:41Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2811, "optime" : { "t" : 1358377780000, "i" : 5404 }, "optimeDate" : ISODate("2013-01-16T23:09:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2811, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2823, "optime" : { "t" : 1358377781000, "i" : 4391 }, "optimeDate" : ISODate("2013-01-16T23:09:41Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:41.774 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:43.377 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:43.774 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2829, "optime" : { "t" : 1358377785000, "i" : 6426 }, "optimeDate" : ISODate("2013-01-16T23:09:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2818, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2818, "optime" : { "t" : 1358377784000, "i" : 93 }, "optimeDate" : ISODate("2013-01-16T23:09:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:44Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2828, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2816, "optime" : { "t" : 1358377781000, "i" : 4497 }, "optimeDate" : ISODate("2013-01-16T23:09:41Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2815, "optime" : { "t" : 1358377784000, "i" : 3311 }, "optimeDate" : ISODate("2013-01-16T23:09:44Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2815, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:44Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2827, "optime" : { "t" : 1358377785000, "i" : 6426 }, "optimeDate" : ISODate("2013-01-16T23:09:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } m31000| Wed Jan 16 18:09:45.774 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31001| Wed Jan 16 18:09:46.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Wed Jan 16 18:09:47.775 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31002| Wed Jan 16 18:09:48.358 [conn171] end connection 127.0.0.3:36979 (3 connections now open) m31002| Wed Jan 16 18:09:48.358 [initandlisten] connection accepted from 127.0.0.3:51132 #173 (4 connections now open) m31002| Wed Jan 16 18:09:48.378 [rsMgr] replSet info two primaries (transiently) m31000| Wed Jan 16 18:09:49.775 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2833, "optime" : { "t" : 1358377789000, "i" : 8031 }, "optimeDate" : ISODate("2013-01-16T23:09:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2822, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2822, "optime" : { "t" : 1358377789000, "i" : 4859 }, "optimeDate" : ISODate("2013-01-16T23:09:49Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:49Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358374960000, "i" : 1 }, "optimeDate" : ISODate("2013-01-16T22:22:40Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2832, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2820, "optime" : { "t" : 1358377785000, "i" : 6426 }, "optimeDate" : ISODate("2013-01-16T23:09:45Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:48Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-16T23:09:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2819, "optime" : { "t" : 1358377788000, "i" : 1715 }, "optimeDate" : ISODate("2013-01-16T23:09:48Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-16T23:09:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 2819, "optime" : { "t" : 1358375223000, "i" : 868 }, "optimeDate" : ISODate("2013-01-16T22:27:03Z"), "lastHeartbeat" : ISODate("2013-01-16T23:09:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 2831, "optime" : { "t" : 1358377789000, "i" : 7703 }, "optimeDate" : ISODate("2013-01-16T23:09:49Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000", "127.0.0.3:31001" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.4:31002" ] } } Wed Jan 16 18:09:52.882 got signal 2 (Interrupt), will terminate after current cmd ends Wed Jan 16 18:09:52.882 [interruptThread] now exiting Wed Jan 16 18:09:52.882 dbexit: Wed Jan 16 18:09:52.883 [interruptThread] shutdown: going to close listening sockets... Wed Jan 16 18:09:52.883 [interruptThread] closing listening socket: 9 Wed Jan 16 18:09:52.883 [interruptThread] closing listening socket: 10 Wed Jan 16 18:09:52.883 [interruptThread] closing listening socket: 11 Wed Jan 16 18:09:52.883 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Wed Jan 16 18:09:52.883 [interruptThread] shutdown: going to flush diaglog... Wed Jan 16 18:09:52.883 [interruptThread] shutdown: going to close sockets... Wed Jan 16 18:09:52.883 [interruptThread] shutdown: waiting for fs preallocator... Wed Jan 16 18:09:52.883 [interruptThread] shutdown: lock for final commit... Wed Jan 16 18:09:52.883 [interruptThread] shutdown: final commit... Wed Jan 16 18:09:52.931 [interruptThread] shutdown: closing all files... Wed Jan 16 18:09:52.932 [interruptThread] closeAllFiles() finished Wed Jan 16 18:09:52.932 [interruptThread] journalCleanup... Wed Jan 16 18:09:52.932 [interruptThread] removeJournalFiles Wed Jan 16 18:09:53.022 [interruptThread] shutdown: removing fs lock... Wed Jan 16 18:09:53.041 dbexit: really exiting now 0 tests succeeded 1 tests didn't get run