cwd [/home/gregorv/Workspaces/10Gen/mongo] num procs:221 removing: /data/db/sconsTests//local.ns removing: /data/db/sconsTests//mongod.lock removing: /data/db/sconsTests//local.0 Fri Jan 18 13:08:29.052 [initandlisten] MongoDB starting : pid=21647 port=27999 dbpath=/data/db/sconsTests/ 64-bit host=nuwen Fri Jan 18 13:08:29.052 [initandlisten] Fri Jan 18 13:08:29.052 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. Fri Jan 18 13:08:29.052 [initandlisten] ** Not recommended for production. Fri Jan 18 13:08:29.052 [initandlisten] Fri Jan 18 13:08:29.052 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 Fri Jan 18 13:08:29.053 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f Fri Jan 18 13:08:29.053 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 Fri Jan 18 13:08:29.053 [initandlisten] allocator: tcmalloc Fri Jan 18 13:08:29.053 [initandlisten] options: { dbpath: "/data/db/sconsTests/", nopreallocj: true, port: 27999, setParameter: [ "enableTestCommands=1" ] } Fri Jan 18 13:08:29.116 [initandlisten] journal dir=/data/db/sconsTests/journal Fri Jan 18 13:08:29.116 [initandlisten] recover : no journal files present, no recovery needed Fri Jan 18 13:08:29.159 [FileAllocator] allocating new datafile /data/db/sconsTests/local.ns, filling with zeroes... Fri Jan 18 13:08:29.159 [FileAllocator] creating directory /data/db/sconsTests/_tmp Fri Jan 18 13:08:29.342 [FileAllocator] done allocating datafile /data/db/sconsTests/local.ns, size: 16MB, took 0.15 secs Fri Jan 18 13:08:29.342 [FileAllocator] allocating new datafile /data/db/sconsTests/local.0, filling with zeroes... Fri Jan 18 13:08:29.876 [FileAllocator] done allocating datafile /data/db/sconsTests/local.0, size: 64MB, took 0.533 secs Fri Jan 18 13:08:29.877 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 718ms Fri Jan 18 13:08:29.877 [initandlisten] waiting for connections on port 27999 Fri Jan 18 13:08:29.878 [websvr] admin web console waiting for connections on port 28999 running /home/gregorv/Workspaces/10Gen/mongo/mongod --port 27999 --dbpath /data/db/sconsTests/ --setParameter enableTestCommands=1 --nopreallocj ******************************************* Test : sync_fast_switch.js ... Command : /home/gregorv/Workspaces/10Gen/mongo/mongo --port 27999 --authenticationMechanism MONGO-CR /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_fast_switch.js --eval TestData = new Object();TestData.testPath = "/home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_fast_switch.js";TestData.testFile = "sync_fast_switch.js";TestData.testName = "sync_fast_switch";TestData.noJournal = false;TestData.noJournalPrealloc = true;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null; Date : Fri Jan 18 13:08:30 2013 MongoDB shell version: 2.3.3-pre- connecting to: 127.0.0.1:27999/test Fri Jan 18 13:08:30.051 [initandlisten] connection accepted from 127.0.0.1:38911 #2 (1 connection now open) null Running: sudo bash -c tc qdisc del root dev lo > /tmp/output Fri Jan 18 13:08:30.056 shell: started program sudo bash -c tc qdisc del root dev lo > /tmp/output Running: sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Fri Jan 18 13:08:30.071 shell: started program sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Running: sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Fri Jan 18 13:08:30.085 shell: started program sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:08:30.098 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002, 31003, 31004 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.2 Resetting db path '/data/db/testReplSet-0' Fri Jan 18 13:08:30.197 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-0 --nopreallocj --bind_ip 127.0.0.2 --setParameter enableTestCommands=1 m31000| note: noprealloc may hurt performance in many applications m31000| Fri Jan 18 13:08:30.224 [initandlisten] MongoDB starting : pid=21689 port=31000 dbpath=/data/db/testReplSet-0 64-bit host=nuwen m31000| Fri Jan 18 13:08:30.224 [initandlisten] m31000| Fri Jan 18 13:08:30.224 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31000| Fri Jan 18 13:08:30.224 [initandlisten] ** Not recommended for production. m31000| Fri Jan 18 13:08:30.224 [initandlisten] m31000| Fri Jan 18 13:08:30.224 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31000| Fri Jan 18 13:08:30.224 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f m31000| Fri Jan 18 13:08:30.224 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31000| Fri Jan 18 13:08:30.224 [initandlisten] allocator: tcmalloc m31000| Fri Jan 18 13:08:30.224 [initandlisten] options: { bind_ip: "127.0.0.2", dbpath: "/data/db/testReplSet-0", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31000, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31000| Fri Jan 18 13:08:30.301 [initandlisten] journal dir=/data/db/testReplSet-0/journal m31000| Fri Jan 18 13:08:30.301 [initandlisten] recover : no journal files present, no recovery needed m31000| Fri Jan 18 13:08:30.352 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.ns, filling with zeroes... m31000| Fri Jan 18 13:08:30.352 [FileAllocator] creating directory /data/db/testReplSet-0/_tmp m31000| Fri Jan 18 13:08:30.535 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.ns, size: 16MB, took 0.149 secs m31000| Fri Jan 18 13:08:30.535 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.0, filling with zeroes... m31000| Fri Jan 18 13:08:30.677 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.0, size: 16MB, took 0.141 secs m31000| Fri Jan 18 13:08:30.678 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 326ms m31000| Fri Jan 18 13:08:30.678 [initandlisten] waiting for connections on port 31000 m31000| Fri Jan 18 13:08:30.678 [websvr] admin web console waiting for connections on port 32000 m31000| Fri Jan 18 13:08:30.680 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Jan 18 13:08:30.680 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Fri Jan 18 13:08:30.799 [initandlisten] connection accepted from 127.0.0.1:38622 #1 (1 connection now open) [ connection to 127.0.0.2:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002, 31003, 31004 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.3 Resetting db path '/data/db/testReplSet-1' Fri Jan 18 13:08:30.883 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-1 --nopreallocj --bind_ip 127.0.0.3 --setParameter enableTestCommands=1 m31001| note: noprealloc may hurt performance in many applications m31001| Fri Jan 18 13:08:30.909 [initandlisten] MongoDB starting : pid=21740 port=31001 dbpath=/data/db/testReplSet-1 64-bit host=nuwen m31001| Fri Jan 18 13:08:30.909 [initandlisten] m31001| Fri Jan 18 13:08:30.909 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31001| Fri Jan 18 13:08:30.909 [initandlisten] ** Not recommended for production. m31001| Fri Jan 18 13:08:30.909 [initandlisten] m31001| Fri Jan 18 13:08:30.909 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31001| Fri Jan 18 13:08:30.909 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f m31001| Fri Jan 18 13:08:30.909 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31001| Fri Jan 18 13:08:30.909 [initandlisten] allocator: tcmalloc m31001| Fri Jan 18 13:08:30.909 [initandlisten] options: { bind_ip: "127.0.0.3", dbpath: "/data/db/testReplSet-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31001, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31001| Fri Jan 18 13:08:30.969 [initandlisten] journal dir=/data/db/testReplSet-1/journal m31001| Fri Jan 18 13:08:30.969 [initandlisten] recover : no journal files present, no recovery needed m31001| Fri Jan 18 13:08:31.020 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.ns, filling with zeroes... m31001| Fri Jan 18 13:08:31.020 [FileAllocator] creating directory /data/db/testReplSet-1/_tmp m31001| Fri Jan 18 13:08:31.196 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.ns, size: 16MB, took 0.142 secs m31001| Fri Jan 18 13:08:31.196 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.0, filling with zeroes... m31001| Fri Jan 18 13:08:31.338 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.0, size: 16MB, took 0.141 secs m31001| Fri Jan 18 13:08:31.339 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 318ms m31001| Fri Jan 18 13:08:31.339 [initandlisten] waiting for connections on port 31001 m31001| Fri Jan 18 13:08:31.340 [websvr] admin web console waiting for connections on port 32001 m31001| Fri Jan 18 13:08:31.341 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Fri Jan 18 13:08:31.341 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Fri Jan 18 13:08:31.485 [initandlisten] connection accepted from 127.0.0.1:41742 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002, 31003, 31004 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.4 Resetting db path '/data/db/testReplSet-2' Fri Jan 18 13:08:31.567 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31002 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-2 --nopreallocj --bind_ip 127.0.0.4 --setParameter enableTestCommands=1 m31002| note: noprealloc may hurt performance in many applications m31002| Fri Jan 18 13:08:31.593 [initandlisten] MongoDB starting : pid=21791 port=31002 dbpath=/data/db/testReplSet-2 64-bit host=nuwen m31002| Fri Jan 18 13:08:31.593 [initandlisten] m31002| Fri Jan 18 13:08:31.593 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31002| Fri Jan 18 13:08:31.593 [initandlisten] ** Not recommended for production. m31002| Fri Jan 18 13:08:31.593 [initandlisten] m31002| Fri Jan 18 13:08:31.593 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31002| Fri Jan 18 13:08:31.593 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f m31002| Fri Jan 18 13:08:31.593 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31002| Fri Jan 18 13:08:31.593 [initandlisten] allocator: tcmalloc m31002| Fri Jan 18 13:08:31.593 [initandlisten] options: { bind_ip: "127.0.0.4", dbpath: "/data/db/testReplSet-2", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31002, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31002| Fri Jan 18 13:08:31.655 [initandlisten] journal dir=/data/db/testReplSet-2/journal m31002| Fri Jan 18 13:08:31.655 [initandlisten] recover : no journal files present, no recovery needed m31002| Fri Jan 18 13:08:31.706 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.ns, filling with zeroes... m31002| Fri Jan 18 13:08:31.706 [FileAllocator] creating directory /data/db/testReplSet-2/_tmp m31002| Fri Jan 18 13:08:31.889 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.ns, size: 16MB, took 0.15 secs m31002| Fri Jan 18 13:08:31.890 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.0, filling with zeroes... m31002| Fri Jan 18 13:08:32.039 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.0, size: 16MB, took 0.149 secs m31002| Fri Jan 18 13:08:32.040 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 334ms m31002| Fri Jan 18 13:08:32.041 [initandlisten] waiting for connections on port 31002 m31002| Fri Jan 18 13:08:32.041 [websvr] admin web console waiting for connections on port 32002 m31002| Fri Jan 18 13:08:32.042 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Fri Jan 18 13:08:32.042 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Fri Jan 18 13:08:32.168 [initandlisten] connection accepted from 127.0.0.1:42642 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002 ] ReplSetTest n is : 3 ReplSetTest n: 3 ports: [ 31000, 31001, 31002, 31003, 31004 ] 31003 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31003, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 3, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.5 Resetting db path '/data/db/testReplSet-3' Fri Jan 18 13:08:32.245 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31003 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-3 --nopreallocj --bind_ip 127.0.0.5 --setParameter enableTestCommands=1 m31003| note: noprealloc may hurt performance in many applications m31003| Fri Jan 18 13:08:32.268 [initandlisten] MongoDB starting : pid=21842 port=31003 dbpath=/data/db/testReplSet-3 64-bit host=nuwen m31003| Fri Jan 18 13:08:32.268 [initandlisten] m31003| Fri Jan 18 13:08:32.268 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31003| Fri Jan 18 13:08:32.268 [initandlisten] ** Not recommended for production. m31003| Fri Jan 18 13:08:32.268 [initandlisten] m31003| Fri Jan 18 13:08:32.268 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31003| Fri Jan 18 13:08:32.268 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f m31003| Fri Jan 18 13:08:32.268 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31003| Fri Jan 18 13:08:32.268 [initandlisten] allocator: tcmalloc m31003| Fri Jan 18 13:08:32.268 [initandlisten] options: { bind_ip: "127.0.0.5", dbpath: "/data/db/testReplSet-3", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31003, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31003| Fri Jan 18 13:08:32.332 [initandlisten] journal dir=/data/db/testReplSet-3/journal m31003| Fri Jan 18 13:08:32.332 [initandlisten] recover : no journal files present, no recovery needed m31003| Fri Jan 18 13:08:32.375 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.ns, filling with zeroes... m31003| Fri Jan 18 13:08:32.375 [FileAllocator] creating directory /data/db/testReplSet-3/_tmp m31003| Fri Jan 18 13:08:32.558 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.ns, size: 16MB, took 0.15 secs m31003| Fri Jan 18 13:08:32.558 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.0, filling with zeroes... m31003| Fri Jan 18 13:08:32.699 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.0, size: 16MB, took 0.141 secs m31003| Fri Jan 18 13:08:32.700 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 325ms m31003| Fri Jan 18 13:08:32.701 [initandlisten] waiting for connections on port 31003 m31003| Fri Jan 18 13:08:32.702 [websvr] admin web console waiting for connections on port 32003 m31003| Fri Jan 18 13:08:32.703 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31003| Fri Jan 18 13:08:32.703 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31003| Fri Jan 18 13:08:32.846 [initandlisten] connection accepted from 127.0.0.1:60063 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002, connection to 127.0.0.5:31003 ] ReplSetTest n is : 4 ReplSetTest n: 4 ports: [ 31000, 31001, 31002, 31003, 31004 ] 31004 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31004, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 4, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.6 Resetting db path '/data/db/testReplSet-4' Fri Jan 18 13:08:32.920 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31004 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-4 --nopreallocj --bind_ip 127.0.0.6 --setParameter enableTestCommands=1 m31004| note: noprealloc may hurt performance in many applications m31004| Fri Jan 18 13:08:32.948 [initandlisten] MongoDB starting : pid=21894 port=31004 dbpath=/data/db/testReplSet-4 64-bit host=nuwen m31004| Fri Jan 18 13:08:32.948 [initandlisten] m31004| Fri Jan 18 13:08:32.948 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31004| Fri Jan 18 13:08:32.948 [initandlisten] ** Not recommended for production. m31004| Fri Jan 18 13:08:32.948 [initandlisten] m31004| Fri Jan 18 13:08:32.948 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31004| Fri Jan 18 13:08:32.948 [initandlisten] git version: d3e8692ad5981bba5eaa50966ba0baf3147e7d9f m31004| Fri Jan 18 13:08:32.948 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31004| Fri Jan 18 13:08:32.948 [initandlisten] allocator: tcmalloc m31004| Fri Jan 18 13:08:32.948 [initandlisten] options: { bind_ip: "127.0.0.6", dbpath: "/data/db/testReplSet-4", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31004, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31004| Fri Jan 18 13:08:33.009 [initandlisten] journal dir=/data/db/testReplSet-4/journal m31004| Fri Jan 18 13:08:33.009 [initandlisten] recover : no journal files present, no recovery needed m31004| Fri Jan 18 13:08:33.060 [FileAllocator] allocating new datafile /data/db/testReplSet-4/local.ns, filling with zeroes... m31004| Fri Jan 18 13:08:33.060 [FileAllocator] creating directory /data/db/testReplSet-4/_tmp m31004| Fri Jan 18 13:08:33.243 [FileAllocator] done allocating datafile /data/db/testReplSet-4/local.ns, size: 16MB, took 0.15 secs m31004| Fri Jan 18 13:08:33.243 [FileAllocator] allocating new datafile /data/db/testReplSet-4/local.0, filling with zeroes... m31004| Fri Jan 18 13:08:33.393 [FileAllocator] done allocating datafile /data/db/testReplSet-4/local.0, size: 16MB, took 0.149 secs m31004| Fri Jan 18 13:08:33.394 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 334ms m31004| Fri Jan 18 13:08:33.394 [initandlisten] waiting for connections on port 31004 m31004| Fri Jan 18 13:08:33.394 [websvr] admin web console waiting for connections on port 32004 m31004| Fri Jan 18 13:08:33.396 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31004| Fri Jan 18 13:08:33.396 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31004| Fri Jan 18 13:08:33.522 [initandlisten] connection accepted from 127.0.0.1:59831 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002, connection to 127.0.0.5:31003, connection to 127.0.0.6:31004 ] { "replSetInitiate" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000" }, { "_id" : 1, "host" : "127.0.0.3:31001" }, { "_id" : 2, "host" : "127.0.0.4:31002" }, { "_id" : 3, "host" : "127.0.0.5:31003" }, { "_id" : 4, "host" : "127.0.0.6:31004" } ] } } m31000| Fri Jan 18 13:08:33.525 [conn1] replSet replSetInitiate admin command received from client m31000| Fri Jan 18 13:08:33.525 [conn1] replSet replSetInitiate config object parses ok, 5 members specified m31001| Fri Jan 18 13:08:33.525 [initandlisten] connection accepted from 127.0.0.2:33311 #2 (2 connections now open) m31002| Fri Jan 18 13:08:33.526 [initandlisten] connection accepted from 127.0.0.2:40222 #2 (2 connections now open) m31003| Fri Jan 18 13:08:33.527 [initandlisten] connection accepted from 127.0.0.2:47307 #2 (2 connections now open) m31004| Fri Jan 18 13:08:33.528 [initandlisten] connection accepted from 127.0.0.2:54430 #2 (2 connections now open) m31000| Fri Jan 18 13:08:33.529 [conn1] replSet replSetInitiate all members seem up m31000| Fri Jan 18 13:08:33.529 [conn1] ****** m31000| Fri Jan 18 13:08:33.529 [conn1] creating replication oplog of size: 40MB... m31000| Fri Jan 18 13:08:33.529 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.1, filling with zeroes... m31000| Fri Jan 18 13:08:34.059 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.1, size: 64MB, took 0.53 secs m31000| Fri Jan 18 13:08:35.127 [conn1] ****** m31000| Fri Jan 18 13:08:35.127 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Jan 18 13:08:35.128 [conn1] replSet saveConfigLocally done m31000| Fri Jan 18 13:08:35.128 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Fri Jan 18 13:08:35.129 [conn1] build index local.replset.minvalid { _id: 1 } m31000| Fri Jan 18 13:08:35.130 [conn1] build index done. scanned 0 total records. 0 secs m31000| Fri Jan 18 13:08:35.130 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000" }, { _id: 1.0, host: "127.0.0.3:31001" }, { _id: 2.0, host: "127.0.0.4:31002" }, { _id: 3.0, host: "127.0.0.5:31003" }, { _id: 4.0, host: "127.0.0.6:31004" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1601133 reslen:112 1605ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Fri Jan 18 13:08:40.681 [rsStart] replSet I am 127.0.0.2:31000 m31000| Fri Jan 18 13:08:40.681 [rsStart] replSet STARTUP2 m31000| Fri Jan 18 13:08:40.681 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31002| Fri Jan 18 13:08:40.681 [conn2] end connection 127.0.0.2:40222 (1 connection now open) m31002| Fri Jan 18 13:08:40.682 [initandlisten] connection accepted from 127.0.0.2:41381 #3 (2 connections now open) m31000| Fri Jan 18 13:08:40.682 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31000| Fri Jan 18 13:08:40.682 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Fri Jan 18 13:08:40.682 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31001| Fri Jan 18 13:08:41.342 [rsStart] trying to contact 127.0.0.2:31000 m31000| Fri Jan 18 13:08:41.342 [initandlisten] connection accepted from 127.0.0.3:43073 #2 (2 connections now open) m31001| Fri Jan 18 13:08:41.343 [rsStart] replSet I am 127.0.0.3:31001 m31001| Fri Jan 18 13:08:41.343 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Fri Jan 18 13:08:41.343 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Fri Jan 18 13:08:41.664 [rsStart] replSet saveConfigLocally done m31001| Fri Jan 18 13:08:41.664 [rsStart] replSet STARTUP2 m31001| Fri Jan 18 13:08:41.665 [rsSync] ****** m31001| Fri Jan 18 13:08:41.665 [rsSync] creating replication oplog of size: 40MB... m31000| Fri Jan 18 13:08:41.682 [rsSync] replSet SECONDARY m31001| Fri Jan 18 13:08:41.698 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.1, filling with zeroes... m31002| Fri Jan 18 13:08:42.043 [rsStart] trying to contact 127.0.0.2:31000 m31000| Fri Jan 18 13:08:42.043 [initandlisten] connection accepted from 127.0.0.4:58639 #3 (3 connections now open) m31002| Fri Jan 18 13:08:42.043 [rsStart] replSet I am 127.0.0.4:31002 m31002| Fri Jan 18 13:08:42.044 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Fri Jan 18 13:08:42.044 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Fri Jan 18 13:08:42.264 [rsStart] replSet saveConfigLocally done m31002| Fri Jan 18 13:08:42.264 [rsStart] replSet STARTUP2 m31001| Fri Jan 18 13:08:42.264 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.1, size: 64MB, took 0.565 secs m31002| Fri Jan 18 13:08:42.264 [rsSync] ****** m31002| Fri Jan 18 13:08:42.264 [rsSync] creating replication oplog of size: 40MB... m31002| Fri Jan 18 13:08:42.506 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.1, filling with zeroes... m31000| Fri Jan 18 13:08:42.682 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31000| Fri Jan 18 13:08:42.682 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31000| Fri Jan 18 13:08:42.682 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Fri Jan 18 13:08:42.682 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31000| Fri Jan 18 13:08:42.683 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31000| Fri Jan 18 13:08:42.683 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31003| Fri Jan 18 13:08:42.704 [rsStart] trying to contact 127.0.0.2:31000 m31000| Fri Jan 18 13:08:42.704 [initandlisten] connection accepted from 127.0.0.5:43245 #4 (4 connections now open) m31003| Fri Jan 18 13:08:42.705 [rsStart] replSet I am 127.0.0.5:31003 m31003| Fri Jan 18 13:08:42.705 [rsStart] replSet got config version 1 from a remote, saving locally m31003| Fri Jan 18 13:08:42.705 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Fri Jan 18 13:08:43.343 [initandlisten] connection accepted from 127.0.0.3:36223 #4 (3 connections now open) m31001| Fri Jan 18 13:08:43.343 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Fri Jan 18 13:08:43.343 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Fri Jan 18 13:08:43.343 [initandlisten] connection accepted from 127.0.0.3:41821 #3 (3 connections now open) m31004| Fri Jan 18 13:08:43.344 [initandlisten] connection accepted from 127.0.0.3:59425 #3 (3 connections now open) m31002| Fri Jan 18 13:08:43.344 [initandlisten] connection accepted from 127.0.0.3:46134 #5 (4 connections now open) m31002| Fri Jan 18 13:08:43.344 [conn4] end connection 127.0.0.3:36223 (3 connections now open) m31001| Fri Jan 18 13:08:43.344 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Fri Jan 18 13:08:43.344 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31001| Fri Jan 18 13:08:43.344 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31001| Fri Jan 18 13:08:43.344 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Fri Jan 18 13:08:43.344 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31001| Fri Jan 18 13:08:43.363 [rsSync] ****** m31001| Fri Jan 18 13:08:43.363 [rsSync] replSet initial sync pending m31001| Fri Jan 18 13:08:43.363 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31002| Fri Jan 18 13:08:43.363 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.1, size: 64MB, took 0.857 secs m31004| Fri Jan 18 13:08:43.396 [rsStart] trying to contact 127.0.0.2:31000 m31000| Fri Jan 18 13:08:43.396 [initandlisten] connection accepted from 127.0.0.6:40380 #5 (5 connections now open) m31004| Fri Jan 18 13:08:43.397 [rsStart] trying to contact 127.0.0.3:31001 m31001| Fri Jan 18 13:08:43.398 [initandlisten] connection accepted from 127.0.0.6:60800 #3 (3 connections now open) m31004| Fri Jan 18 13:08:43.398 [rsStart] replSet I am 127.0.0.6:31004 m31004| Fri Jan 18 13:08:43.399 [rsStart] replSet got config version 1 from a remote, saving locally m31004| Fri Jan 18 13:08:43.399 [rsStart] replSet info saving a newer config version to local.system.replset m31003| Fri Jan 18 13:08:43.401 [rsStart] replSet saveConfigLocally done m31003| Fri Jan 18 13:08:43.402 [rsStart] replSet STARTUP2 m31003| Fri Jan 18 13:08:43.402 [rsSync] ****** m31003| Fri Jan 18 13:08:43.402 [rsSync] creating replication oplog of size: 40MB... m31004| Fri Jan 18 13:08:43.814 [rsStart] replSet saveConfigLocally done m31004| Fri Jan 18 13:08:43.814 [rsStart] replSet STARTUP2 m31003| Fri Jan 18 13:08:43.815 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.1, filling with zeroes... m31004| Fri Jan 18 13:08:43.815 [rsSync] ****** m31004| Fri Jan 18 13:08:43.815 [rsSync] creating replication oplog of size: 40MB... m31004| Fri Jan 18 13:08:44.044 [initandlisten] connection accepted from 127.0.0.4:59959 #4 (4 connections now open) m31003| Fri Jan 18 13:08:44.044 [initandlisten] connection accepted from 127.0.0.4:38714 #4 (4 connections now open) m31001| Fri Jan 18 13:08:44.044 [initandlisten] connection accepted from 127.0.0.4:50567 #4 (4 connections now open) m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Fri Jan 18 13:08:44.044 [initandlisten] connection accepted from 127.0.0.4:34636 #5 (5 connections now open) m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replset info 127.0.0.6:31004 thinks that we are down m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state STARTUP2 m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replset info 127.0.0.5:31003 thinks that we are down m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31001| Fri Jan 18 13:08:44.044 [conn4] end connection 127.0.0.4:50567 (4 connections now open) m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Fri Jan 18 13:08:44.044 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31004| Fri Jan 18 13:08:44.464 [FileAllocator] allocating new datafile /data/db/testReplSet-4/local.1, filling with zeroes... m31002| Fri Jan 18 13:08:44.480 [rsSync] ****** m31002| Fri Jan 18 13:08:44.480 [rsSync] replSet initial sync pending m31003| Fri Jan 18 13:08:44.480 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.1, size: 64MB, took 0.665 secs m31002| Fri Jan 18 13:08:44.480 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Fri Jan 18 13:08:44.682 [rsHealthPoll] replset info 127.0.0.6:31004 thinks that we are down m31000| Fri Jan 18 13:08:44.682 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state STARTUP2 m31000| Fri Jan 18 13:08:44.682 [rsMgr] not electing self, 127.0.0.6:31004 would veto with 'I don't think 127.0.0.2:31000 is electable' m31000| Fri Jan 18 13:08:44.682 [rsHealthPoll] replset info 127.0.0.5:31003 thinks that we are down m31000| Fri Jan 18 13:08:44.682 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31000| Fri Jan 18 13:08:44.683 [rsMgr] not electing self, 127.0.0.6:31004 would veto with 'I don't think 127.0.0.2:31000 is electable' m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31004| Fri Jan 18 13:08:44.705 [initandlisten] connection accepted from 127.0.0.5:49129 #5 (5 connections now open) m31001| Fri Jan 18 13:08:44.705 [initandlisten] connection accepted from 127.0.0.5:49249 #6 (5 connections now open) m31002| Fri Jan 18 13:08:44.705 [initandlisten] connection accepted from 127.0.0.5:60207 #6 (4 connections now open) m31001| Fri Jan 18 13:08:44.705 [conn6] end connection 127.0.0.5:49249 (4 connections now open) m31001| Fri Jan 18 13:08:44.705 [initandlisten] connection accepted from 127.0.0.5:42252 #7 (6 connections now open) m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replset info 127.0.0.6:31004 thinks that we are down m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state STARTUP2 m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Fri Jan 18 13:08:44.705 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31001| Fri Jan 18 13:08:45.344 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31001| Fri Jan 18 13:08:45.344 [rsHealthPoll] replset info 127.0.0.6:31004 thinks that we are down m31001| Fri Jan 18 13:08:45.344 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state STARTUP2 m31003| Fri Jan 18 13:08:45.389 [rsSync] ****** m31003| Fri Jan 18 13:08:45.389 [rsSync] replSet initial sync pending m31003| Fri Jan 18 13:08:45.389 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31004| Fri Jan 18 13:08:45.389 [FileAllocator] done allocating datafile /data/db/testReplSet-4/local.1, size: 64MB, took 0.924 secs m31001| Fri Jan 18 13:08:45.399 [conn3] end connection 127.0.0.6:60800 (4 connections now open) m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Fri Jan 18 13:08:45.399 [initandlisten] connection accepted from 127.0.0.6:50835 #7 (5 connections now open) m31003| Fri Jan 18 13:08:45.399 [initandlisten] connection accepted from 127.0.0.6:38850 #5 (5 connections now open) m31001| Fri Jan 18 13:08:45.399 [initandlisten] connection accepted from 127.0.0.6:47685 #8 (5 connections now open) m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31004| Fri Jan 18 13:08:45.399 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31004| Fri Jan 18 13:08:45.400 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31004| Fri Jan 18 13:08:46.322 [rsSync] ****** m31004| Fri Jan 18 13:08:46.322 [rsSync] replSet initial sync pending m31004| Fri Jan 18 13:08:46.322 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Fri Jan 18 13:08:50.683 [rsMgr] replSet info electSelf 0 m31004| Fri Jan 18 13:08:50.684 [conn2] replSet RECOVERING m31004| Fri Jan 18 13:08:50.684 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31002| Fri Jan 18 13:08:50.684 [conn3] replSet RECOVERING m31002| Fri Jan 18 13:08:50.684 [conn3] replSet info voting yea for 127.0.0.2:31000 (0) m31003| Fri Jan 18 13:08:50.684 [conn2] replSet RECOVERING m31003| Fri Jan 18 13:08:50.684 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Fri Jan 18 13:08:50.684 [conn2] replSet RECOVERING m31001| Fri Jan 18 13:08:50.684 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31003| Fri Jan 18 13:08:50.706 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state RECOVERING m31003| Fri Jan 18 13:08:50.706 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31003| Fri Jan 18 13:08:50.706 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31001| Fri Jan 18 13:08:51.345 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31001| Fri Jan 18 13:08:51.345 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state RECOVERING m31001| Fri Jan 18 13:08:51.345 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31004| Fri Jan 18 13:08:51.400 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31004| Fri Jan 18 13:08:51.401 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31004| Fri Jan 18 13:08:51.401 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31000| Fri Jan 18 13:08:51.683 [rsMgr] replSet PRIMARY m31002| Fri Jan 18 13:08:52.045 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Fri Jan 18 13:08:52.046 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31002| Fri Jan 18 13:08:52.046 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state RECOVERING m31002| Fri Jan 18 13:08:52.046 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31000| Fri Jan 18 13:08:52.683 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state RECOVERING m31000| Fri Jan 18 13:08:52.683 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31000| Fri Jan 18 13:08:52.684 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31000| Fri Jan 18 13:08:52.684 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31003| Fri Jan 18 13:08:52.706 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Fri Jan 18 13:08:53.345 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31004| Fri Jan 18 13:08:53.400 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31003| Fri Jan 18 13:08:54.684 [conn2] end connection 127.0.0.2:47307 (4 connections now open) m31003| Fri Jan 18 13:08:54.684 [initandlisten] connection accepted from 127.0.0.2:33062 #6 (5 connections now open) m31001| Fri Jan 18 13:08:56.685 [conn2] end connection 127.0.0.2:33311 (4 connections now open) m31001| Fri Jan 18 13:08:56.685 [initandlisten] connection accepted from 127.0.0.2:48196 #9 (5 connections now open) m31003| Fri Jan 18 13:08:57.346 [conn3] end connection 127.0.0.3:41821 (4 connections now open) m31003| Fri Jan 18 13:08:57.346 [initandlisten] connection accepted from 127.0.0.3:52746 #7 (5 connections now open) m31003| Fri Jan 18 13:08:58.046 [conn4] end connection 127.0.0.4:38714 (4 connections now open) m31003| Fri Jan 18 13:08:58.047 [initandlisten] connection accepted from 127.0.0.4:48915 #8 (5 connections now open) m31002| Fri Jan 18 13:08:58.708 [conn6] end connection 127.0.0.5:60207 (4 connections now open) m31002| Fri Jan 18 13:08:58.708 [initandlisten] connection accepted from 127.0.0.5:40568 #8 (5 connections now open) m31000| Fri Jan 18 13:08:59.346 [conn2] end connection 127.0.0.3:43073 (4 connections now open) m31000| Fri Jan 18 13:08:59.346 [initandlisten] connection accepted from 127.0.0.3:58544 #6 (6 connections now open) m31001| Fri Jan 18 13:08:59.364 [rsSync] replSet initial sync pending m31001| Fri Jan 18 13:08:59.364 [rsSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31001| Fri Jan 18 13:08:59.364 [rsSync] SYNCED TO 127.0.0.2:31000 m31001| Fri Jan 18 13:08:59.364 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:08:59.364 [initandlisten] connection accepted from 127.0.0.3:32931 #7 (6 connections now open) m31001| Fri Jan 18 13:08:59.365 [rsSync] build index local.me { _id: 1 } m31001| Fri Jan 18 13:08:59.366 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Fri Jan 18 13:08:59.366 [rsSync] replSet initial sync drop all databases m31001| Fri Jan 18 13:08:59.367 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Fri Jan 18 13:08:59.367 [rsSync] replSet initial sync clone all databases m31001| Fri Jan 18 13:08:59.367 [rsSync] replSet initial sync data copy, starting syncup m31001| Fri Jan 18 13:08:59.367 [rsSync] oplog sync 1 of 3 m31001| Fri Jan 18 13:08:59.367 [rsSync] oplog sync 2 of 3 m31001| Fri Jan 18 13:08:59.368 [rsSync] replSet initial sync building indexes m31001| Fri Jan 18 13:08:59.368 [rsSync] oplog sync 3 of 3 m31001| Fri Jan 18 13:08:59.368 [rsSync] replSet initial sync finishing up m31001| Fri Jan 18 13:08:59.368 [rsSync] replSet set minValid=50f98fa3:1 m31001| Fri Jan 18 13:08:59.369 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Fri Jan 18 13:08:59.369 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Fri Jan 18 13:08:59.370 [rsSync] replSet initial sync done m31000| Fri Jan 18 13:08:59.370 [conn7] end connection 127.0.0.3:32931 (5 connections now open) m31002| Fri Jan 18 13:08:59.402 [conn7] end connection 127.0.0.6:50835 (4 connections now open) m31002| Fri Jan 18 13:08:59.402 [initandlisten] connection accepted from 127.0.0.6:33054 #9 (5 connections now open) m31001| Fri Jan 18 13:08:59.666 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31001| Fri Jan 18 13:08:59.666 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31001| Fri Jan 18 13:08:59.666 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:08:59.666 [initandlisten] connection accepted from 127.0.0.3:36955 #8 (6 connections now open) m31000| Fri Jan 18 13:09:00.047 [conn3] end connection 127.0.0.4:58639 (5 connections now open) m31000| Fri Jan 18 13:09:00.047 [initandlisten] connection accepted from 127.0.0.4:51412 #9 (6 connections now open) m31001| Fri Jan 18 13:09:00.370 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:00.370 [initandlisten] connection accepted from 127.0.0.3:54335 #10 (7 connections now open) m31002| Fri Jan 18 13:09:00.481 [rsSync] replSet initial sync pending m31002| Fri Jan 18 13:09:00.481 [rsSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31002| Fri Jan 18 13:09:00.481 [rsSync] SYNCED TO 127.0.0.2:31000 m31002| Fri Jan 18 13:09:00.481 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:00.481 [initandlisten] connection accepted from 127.0.0.4:34111 #11 (8 connections now open) m31002| Fri Jan 18 13:09:00.482 [rsSync] build index local.me { _id: 1 } m31002| Fri Jan 18 13:09:00.483 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Fri Jan 18 13:09:00.483 [rsSync] replSet initial sync drop all databases m31002| Fri Jan 18 13:09:00.483 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Fri Jan 18 13:09:00.483 [rsSync] replSet initial sync clone all databases m31002| Fri Jan 18 13:09:00.484 [rsSync] replSet initial sync data copy, starting syncup m31002| Fri Jan 18 13:09:00.484 [rsSync] oplog sync 1 of 3 m31002| Fri Jan 18 13:09:00.484 [rsSync] oplog sync 2 of 3 m31002| Fri Jan 18 13:09:00.484 [rsSync] replSet initial sync building indexes m31002| Fri Jan 18 13:09:00.484 [rsSync] oplog sync 3 of 3 m31002| Fri Jan 18 13:09:00.485 [rsSync] replSet initial sync finishing up m31002| Fri Jan 18 13:09:00.485 [rsSync] replSet set minValid=50f98fa3:1 m31002| Fri Jan 18 13:09:00.486 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Fri Jan 18 13:09:00.486 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Fri Jan 18 13:09:00.487 [rsSync] replSet initial sync done m31000| Fri Jan 18 13:09:00.487 [conn11] end connection 127.0.0.4:34111 (7 connections now open) m31000| Fri Jan 18 13:09:00.707 [conn4] end connection 127.0.0.5:43245 (6 connections now open) m31000| Fri Jan 18 13:09:00.707 [initandlisten] connection accepted from 127.0.0.5:49835 #12 (7 connections now open) m31002| Fri Jan 18 13:09:01.266 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31002| Fri Jan 18 13:09:01.266 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31002| Fri Jan 18 13:09:01.266 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:01.266 [initandlisten] connection accepted from 127.0.0.4:56288 #13 (8 connections now open) m31001| Fri Jan 18 13:09:01.370 [rsSync] replSet SECONDARY m31000| Fri Jan 18 13:09:01.372 [slaveTracking] build index local.slaves { _id: 1 } m31000| Fri Jan 18 13:09:01.372 [slaveTracking] build index done. scanned 0 total records. 0 secs m31003| Fri Jan 18 13:09:01.389 [rsSync] replSet initial sync pending m31003| Fri Jan 18 13:09:01.389 [rsSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31003| Fri Jan 18 13:09:01.389 [rsSync] SYNCED TO 127.0.0.2:31000 m31003| Fri Jan 18 13:09:01.389 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:01.390 [initandlisten] connection accepted from 127.0.0.5:37079 #14 (9 connections now open) m31003| Fri Jan 18 13:09:01.391 [rsSync] build index local.me { _id: 1 } m31003| Fri Jan 18 13:09:01.391 [rsSync] build index done. scanned 0 total records. 0 secs m31003| Fri Jan 18 13:09:01.392 [rsSync] replSet initial sync drop all databases m31003| Fri Jan 18 13:09:01.392 [rsSync] dropAllDatabasesExceptLocal 1 m31003| Fri Jan 18 13:09:01.392 [rsSync] replSet initial sync clone all databases m31003| Fri Jan 18 13:09:01.392 [rsSync] replSet initial sync data copy, starting syncup m31003| Fri Jan 18 13:09:01.392 [rsSync] oplog sync 1 of 3 m31003| Fri Jan 18 13:09:01.393 [rsSync] oplog sync 2 of 3 m31003| Fri Jan 18 13:09:01.393 [rsSync] replSet initial sync building indexes m31003| Fri Jan 18 13:09:01.393 [rsSync] oplog sync 3 of 3 m31003| Fri Jan 18 13:09:01.393 [rsSync] replSet initial sync finishing up m31003| Fri Jan 18 13:09:01.393 [rsSync] replSet set minValid=50f98fa3:1 m31003| Fri Jan 18 13:09:01.394 [rsSync] build index local.replset.minvalid { _id: 1 } m31003| Fri Jan 18 13:09:01.394 [rsSync] build index done. scanned 0 total records. 0 secs m31003| Fri Jan 18 13:09:01.395 [rsSync] replSet initial sync done m31000| Fri Jan 18 13:09:01.395 [conn14] end connection 127.0.0.5:37079 (8 connections now open) m31000| Fri Jan 18 13:09:01.402 [conn5] end connection 127.0.0.6:40380 (7 connections now open) m31000| Fri Jan 18 13:09:01.402 [initandlisten] connection accepted from 127.0.0.6:53999 #15 (8 connections now open) m31004| Fri Jan 18 13:09:01.402 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Fri Jan 18 13:09:01.404 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31003| Fri Jan 18 13:09:01.404 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31003| Fri Jan 18 13:09:01.404 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:01.404 [initandlisten] connection accepted from 127.0.0.5:37866 #16 (9 connections now open) m31002| Fri Jan 18 13:09:01.487 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:01.487 [initandlisten] connection accepted from 127.0.0.4:56708 #17 (10 connections now open) m31002| Fri Jan 18 13:09:02.047 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31004| Fri Jan 18 13:09:02.323 [rsSync] replSet initial sync pending m31004| Fri Jan 18 13:09:02.323 [rsSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31004| Fri Jan 18 13:09:02.323 [rsSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31004| Fri Jan 18 13:09:02.323 [rsSync] SYNCED TO 127.0.0.2:31000 m31004| Fri Jan 18 13:09:02.323 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:02.323 [initandlisten] connection accepted from 127.0.0.6:60246 #18 (11 connections now open) m31004| Fri Jan 18 13:09:02.324 [rsSync] build index local.me { _id: 1 } m31004| Fri Jan 18 13:09:02.325 [rsSync] build index done. scanned 0 total records. 0 secs m31004| Fri Jan 18 13:09:02.325 [rsSync] replSet initial sync drop all databases m31004| Fri Jan 18 13:09:02.325 [rsSync] dropAllDatabasesExceptLocal 1 m31004| Fri Jan 18 13:09:02.325 [rsSync] replSet initial sync clone all databases m31004| Fri Jan 18 13:09:02.326 [rsSync] replSet initial sync data copy, starting syncup m31004| Fri Jan 18 13:09:02.326 [rsSync] oplog sync 1 of 3 m31004| Fri Jan 18 13:09:02.326 [rsSync] oplog sync 2 of 3 m31004| Fri Jan 18 13:09:02.326 [rsSync] replSet initial sync building indexes m31004| Fri Jan 18 13:09:02.326 [rsSync] oplog sync 3 of 3 m31004| Fri Jan 18 13:09:02.326 [rsSync] replSet initial sync finishing up m31004| Fri Jan 18 13:09:02.327 [rsSync] replSet set minValid=50f98fa3:1 m31004| Fri Jan 18 13:09:02.327 [rsSync] build index local.replset.minvalid { _id: 1 } m31004| Fri Jan 18 13:09:02.328 [rsSync] build index done. scanned 0 total records. 0 secs m31004| Fri Jan 18 13:09:02.328 [rsSync] replSet initial sync done m31000| Fri Jan 18 13:09:02.328 [conn18] end connection 127.0.0.6:60246 (10 connections now open) m31003| Fri Jan 18 13:09:02.395 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:02.395 [initandlisten] connection accepted from 127.0.0.5:57382 #19 (11 connections now open) m31002| Fri Jan 18 13:09:02.487 [rsSync] replSet SECONDARY m31000| Fri Jan 18 13:09:02.685 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Fri Jan 18 13:09:02.686 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Fri Jan 18 13:09:02.708 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Fri Jan 18 13:09:02.709 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31004| Fri Jan 18 13:09:02.816 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31004| Fri Jan 18 13:09:02.816 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31004| Fri Jan 18 13:09:02.816 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:02.816 [initandlisten] connection accepted from 127.0.0.6:47285 #20 (12 connections now open) m31004| Fri Jan 18 13:09:03.328 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:03.328 [initandlisten] connection accepted from 127.0.0.6:38374 #21 (13 connections now open) m31001| Fri Jan 18 13:09:03.347 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Fri Jan 18 13:09:03.395 [rsSync] replSet SECONDARY m31004| Fri Jan 18 13:09:03.402 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31004| Fri Jan 18 13:09:03.403 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31002| Fri Jan 18 13:09:04.048 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31004| Fri Jan 18 13:09:04.329 [rsSync] replSet SECONDARY ---- Started set! ---- { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0, "slaveDelay" : 80 }, { "_id" : 4, "host" : "127.0.0.6:31004", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0, "slaveDelay" : 80 }, { "_id" : 4, "host" : "127.0.0.6:31004", "priority" : 0 } ], "version" : 2 } } m31000| Fri Jan 18 13:09:04.408 [conn1] replSet replSetReconfig config object parses ok, 5 members specified m31000| Fri Jan 18 13:09:04.409 [conn1] replSet replSetReconfig [2] m31000| Fri Jan 18 13:09:04.409 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Jan 18 13:09:04.410 [conn1] replSet saveConfigLocally done m31000| Fri Jan 18 13:09:04.410 [conn1] replSet relinquishing primary state m31000| Fri Jan 18 13:09:04.410 [conn1] replSet SECONDARY m31000| Fri Jan 18 13:09:04.410 [conn1] replSet closing client sockets after relinquishing primary m31000| Fri Jan 18 13:09:04.410 [conn1] replSet PRIMARY Fri Jan 18 13:09:04.410 DBClientCursor::init call() failed m31001| Fri Jan 18 13:09:04.410 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31004| Fri Jan 18 13:09:04.410 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31003| Fri Jan 18 13:09:04.410 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31003| Fri Jan 18 13:09:04.410 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31002| Fri Jan 18 13:09:04.410 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31002| Fri Jan 18 13:09:04.410 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:04.410 [conn1] replSet replSetReconfig new config saved locally m31000| Fri Jan 18 13:09:04.410 [conn20] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.6:47285] m31000| Fri Jan 18 13:09:04.410 [conn19] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.5:57382] m31004| Fri Jan 18 13:09:04.410 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:04.410 [conn13] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:56288] m31000| Fri Jan 18 13:09:04.410 [conn1] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:38622] m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31000| Fri Jan 18 13:09:04.410 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Fri Jan 18 13:09:04.410 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:04.410 [rsMgr] replSet relinquishing primary state m31000| Fri Jan 18 13:09:04.410 [conn21] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.6:38374] m31000| Fri Jan 18 13:09:04.410 [rsMgr] replSet SECONDARY m31000| Fri Jan 18 13:09:04.410 [conn16] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.5:37866] m31000| Fri Jan 18 13:09:04.410 [rsMgr] replSet closing client sockets after relinquishing primary m31000| Fri Jan 18 13:09:04.410 [conn17] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:56708] Fri Jan 18 13:09:04.410 trying reconnect to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Fri Jan 18 13:09:04.410 [conn10] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:54335] m31000| Fri Jan 18 13:09:04.410 [conn8] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:36955] m31000| Fri Jan 18 13:09:04.410 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY Fri Jan 18 13:09:04.411 reconnect 127.0.0.2:31000 ok m31000| Fri Jan 18 13:09:04.411 [initandlisten] connection accepted from 127.0.0.1:38644 #22 (5 connections now open) m31000| Fri Jan 18 13:09:04.411 [rsMgr] not electing self, 127.0.0.6:31004 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Fri Jan 18 13:09:04.411 [rsMgr] not electing self, 127.0.0.6:31004 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Fri Jan 18 13:09:04.411 [rsMgr] not electing self, 127.0.0.6:31004 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Fri Jan 18 13:09:04.685 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Fri Jan 18 13:09:04.685 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Fri Jan 18 13:09:04.686 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Fri Jan 18 13:09:04.686 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Fri Jan 18 13:09:04.708 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Fri Jan 18 13:09:04.708 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31003| Fri Jan 18 13:09:04.709 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31003| Fri Jan 18 13:09:04.709 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Fri Jan 18 13:09:04.853 [rsMgr] replSet saveConfigLocally done m31003| Fri Jan 18 13:09:04.853 [rsMgr] replSet replSetReconfig new config saved locally m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Fri Jan 18 13:09:04.853 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.347 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.348 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.348 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31001| Fri Jan 18 13:09:05.348 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Fri Jan 18 13:09:05.348 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.366 [rsMgr] replSet saveConfigLocally done m31001| Fri Jan 18 13:09:05.366 [rsMgr] replSet replSetReconfig new config saved locally m31001| Fri Jan 18 13:09:05.366 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31001| Fri Jan 18 13:09:05.366 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Fri Jan 18 13:09:05.366 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Fri Jan 18 13:09:05.366 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31004| Fri Jan 18 13:09:05.403 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31004| Fri Jan 18 13:09:05.403 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31004| Fri Jan 18 13:09:05.403 [rsMgr] replSet info saving a newer config version to local.system.replset m31004| Fri Jan 18 13:09:05.415 [rsMgr] replSet saveConfigLocally done m31004| Fri Jan 18 13:09:05.415 [rsMgr] replSet replSetReconfig new config saved locally m31004| Fri Jan 18 13:09:05.415 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31004| Fri Jan 18 13:09:05.415 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31004| Fri Jan 18 13:09:05.415 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31004| Fri Jan 18 13:09:05.415 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31004| Fri Jan 18 13:09:05.415 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.048 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.048 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31002| Fri Jan 18 13:09:06.048 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Fri Jan 18 13:09:06.048 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.063 [rsMgr] replSet saveConfigLocally done m31002| Fri Jan 18 13:09:06.063 [rsMgr] replSet replSetReconfig new config saved locally m31002| Fri Jan 18 13:09:06.063 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31002| Fri Jan 18 13:09:06.064 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31002| Fri Jan 18 13:09:06.064 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Fri Jan 18 13:09:06.064 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.6:31004 is up m31002| Fri Jan 18 13:09:06.064 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.6:31004 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.064 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Fri Jan 18 13:09:06.064 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31004| Fri Jan 18 13:09:06.411 [conn2] end connection 127.0.0.2:54430 (4 connections now open) m31004| Fri Jan 18 13:09:06.411 [initandlisten] connection accepted from 127.0.0.2:52466 #6 (5 connections now open) m31003| Fri Jan 18 13:09:06.709 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Fri Jan 18 13:09:06.709 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Fri Jan 18 13:09:06.709 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Fri Jan 18 13:09:06.709 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:06.853 [conn5] end connection 127.0.0.5:49129 (4 connections now open) m31004| Fri Jan 18 13:09:06.853 [initandlisten] connection accepted from 127.0.0.5:37054 #7 (5 connections now open) m31001| Fri Jan 18 13:09:07.348 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Fri Jan 18 13:09:07.348 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Fri Jan 18 13:09:07.348 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Fri Jan 18 13:09:07.348 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:07.367 [conn3] end connection 127.0.0.3:59425 (4 connections now open) m31004| Fri Jan 18 13:09:07.367 [initandlisten] connection accepted from 127.0.0.3:59703 #8 (5 connections now open) m31004| Fri Jan 18 13:09:07.403 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:07.403 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:07.403 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:07.403 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Fri Jan 18 13:09:07.415 [conn5] end connection 127.0.0.6:38850 (4 connections now open) m31003| Fri Jan 18 13:09:07.415 [initandlisten] connection accepted from 127.0.0.6:48441 #9 (6 connections now open) m31002| Fri Jan 18 13:09:08.048 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Fri Jan 18 13:09:08.048 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Fri Jan 18 13:09:08.048 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Fri Jan 18 13:09:08.048 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31004| Fri Jan 18 13:09:08.064 [conn4] end connection 127.0.0.4:59959 (4 connections now open) m31004| Fri Jan 18 13:09:08.064 [initandlisten] connection accepted from 127.0.0.4:39569 #9 (6 connections now open) m31002| Fri Jan 18 13:09:08.411 [conn3] end connection 127.0.0.2:41381 (4 connections now open) m31002| Fri Jan 18 13:09:08.411 [initandlisten] connection accepted from 127.0.0.2:54078 #10 (5 connections now open) m31001| Fri Jan 18 13:09:08.854 [conn7] end connection 127.0.0.5:42252 (4 connections now open) m31001| Fri Jan 18 13:09:08.854 [initandlisten] connection accepted from 127.0.0.5:51714 #10 (5 connections now open) m31002| Fri Jan 18 13:09:09.367 [conn5] end connection 127.0.0.3:46134 (4 connections now open) m31002| Fri Jan 18 13:09:09.367 [initandlisten] connection accepted from 127.0.0.3:45916 #11 (5 connections now open) m31001| Fri Jan 18 13:09:09.416 [conn8] end connection 127.0.0.6:47685 (4 connections now open) m31001| Fri Jan 18 13:09:09.416 [initandlisten] connection accepted from 127.0.0.6:33390 #11 (5 connections now open) m31001| Fri Jan 18 13:09:10.064 [conn5] end connection 127.0.0.4:34636 (4 connections now open) m31001| Fri Jan 18 13:09:10.064 [initandlisten] connection accepted from 127.0.0.4:53088 #12 (5 connections now open) m31000| Fri Jan 18 13:09:10.412 [rsMgr] replSet info electSelf 0 m31004| Fri Jan 18 13:09:10.412 [conn6] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Fri Jan 18 13:09:10.412 [conn9] replSet info voting yea for 127.0.0.2:31000 (0) m31002| Fri Jan 18 13:09:10.412 [conn10] replSet info voting yea for 127.0.0.2:31000 (0) m31003| Fri Jan 18 13:09:10.412 [conn6] replSet info voting yea for 127.0.0.2:31000 (0) m31000| Fri Jan 18 13:09:10.685 [rsMgr] replSet PRIMARY m31003| Fri Jan 18 13:09:10.854 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31003| Fri Jan 18 13:09:10.854 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Fri Jan 18 13:09:11.367 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31004| Fri Jan 18 13:09:11.416 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Fri Jan 18 13:09:12.065 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31000| Fri Jan 18 13:09:12.425 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.ns, filling with zeroes... My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31000| Fri Jan 18 13:09:12.582 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.ns, size: 16MB, took 0.156 secs m31000| Fri Jan 18 13:09:12.582 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.0, filling with zeroes... m31000| Fri Jan 18 13:09:12.732 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.0, size: 16MB, took 0.149 secs m31000| Fri Jan 18 13:09:12.733 [conn22] build index foo.bar { _id: 1 } m31000| Fri Jan 18 13:09:12.734 [conn22] build index done. scanned 0 total records. 0 secs m31000| Fri Jan 18 13:09:12.734 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:309160 308ms m31002| Fri Jan 18 13:09:14.410 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31002| Fri Jan 18 13:09:14.410 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31002| Fri Jan 18 13:09:14.410 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31003| Fri Jan 18 13:09:14.410 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31003| Fri Jan 18 13:09:14.410 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31003| Fri Jan 18 13:09:14.410 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31001| Fri Jan 18 13:09:14.410 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31001| Fri Jan 18 13:09:14.410 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31001| Fri Jan 18 13:09:14.410 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31004| Fri Jan 18 13:09:14.411 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.2:31000 m31004| Fri Jan 18 13:09:14.411 [rsBackgroundSync] SYNCED TO 127.0.0.2:31000 m31004| Fri Jan 18 13:09:14.411 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Fri Jan 18 13:09:14.411 [initandlisten] connection accepted from 127.0.0.4:52481 #23 (6 connections now open) m31000| Fri Jan 18 13:09:14.411 [initandlisten] connection accepted from 127.0.0.5:45311 #24 (7 connections now open) m31000| Fri Jan 18 13:09:14.411 [initandlisten] connection accepted from 127.0.0.3:57659 #25 (8 connections now open) m31000| Fri Jan 18 13:09:14.411 [initandlisten] connection accepted from 127.0.0.6:48015 #26 (9 connections now open) m31003| Fri Jan 18 13:09:14.411 [rsSync] replSet slavedelay sleep long time: 70 m31004| Fri Jan 18 13:09:14.412 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:14.412 [initandlisten] connection accepted from 127.0.0.6:55643 #27 (10 connections now open) m31004| Fri Jan 18 13:09:14.413 [FileAllocator] allocating new datafile /data/db/testReplSet-4/foo.ns, filling with zeroes... m31004| Fri Jan 18 13:09:14.564 [FileAllocator] done allocating datafile /data/db/testReplSet-4/foo.ns, size: 16MB, took 0.151 secs m31004| Fri Jan 18 13:09:14.565 [FileAllocator] allocating new datafile /data/db/testReplSet-4/foo.0, filling with zeroes... m31004| Fri Jan 18 13:09:14.714 [FileAllocator] done allocating datafile /data/db/testReplSet-4/foo.0, size: 16MB, took 0.149 secs m31004| Fri Jan 18 13:09:14.716 [repl writer worker 1] build index foo.bar { _id: 1 } m31004| Fri Jan 18 13:09:14.716 [repl writer worker 1] build index done. scanned 0 total records. 0 secs My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31003| Fri Jan 18 13:09:22.414 [conn6] end connection 127.0.0.2:33062 (4 connections now open) m31003| Fri Jan 18 13:09:22.414 [initandlisten] connection accepted from 127.0.0.2:60302 #10 (5 connections now open) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31002| Fri Jan 18 13:09:22.856 [conn8] end connection 127.0.0.5:40568 (4 connections now open) m31002| Fri Jan 18 13:09:22.857 [initandlisten] connection accepted from 127.0.0.5:60619 #12 (5 connections now open) m31003| Fri Jan 18 13:09:23.370 [conn7] end connection 127.0.0.3:52746 (4 connections now open) m31003| Fri Jan 18 13:09:23.370 [initandlisten] connection accepted from 127.0.0.3:57511 #11 (6 connections now open) m31002| Fri Jan 18 13:09:23.418 [conn9] end connection 127.0.0.6:33054 (4 connections now open) m31002| Fri Jan 18 13:09:23.418 [initandlisten] connection accepted from 127.0.0.6:60405 #13 (5 connections now open) m31003| Fri Jan 18 13:09:24.067 [conn8] end connection 127.0.0.4:48915 (4 connections now open) m31003| Fri Jan 18 13:09:24.067 [initandlisten] connection accepted from 127.0.0.4:58937 #12 (5 connections now open) m31001| Fri Jan 18 13:09:24.414 [conn9] end connection 127.0.0.2:48196 (4 connections now open) m31001| Fri Jan 18 13:09:24.414 [initandlisten] connection accepted from 127.0.0.2:38379 #13 (5 connections now open) m31000| Fri Jan 18 13:09:24.857 [conn12] end connection 127.0.0.5:49835 (9 connections now open) m31000| Fri Jan 18 13:09:24.857 [initandlisten] connection accepted from 127.0.0.5:35703 #28 (10 connections now open) m31000| Fri Jan 18 13:09:25.370 [conn6] end connection 127.0.0.3:58544 (9 connections now open) m31000| Fri Jan 18 13:09:25.370 [initandlisten] connection accepted from 127.0.0.3:34166 #29 (10 connections now open) m31000| Fri Jan 18 13:09:25.418 [conn15] end connection 127.0.0.6:53999 (9 connections now open) m31000| Fri Jan 18 13:09:25.419 [initandlisten] connection accepted from 127.0.0.6:44779 #30 (10 connections now open) m31000| Fri Jan 18 13:09:26.067 [conn9] end connection 127.0.0.4:51412 (9 connections now open) m31000| Fri Jan 18 13:09:26.068 [initandlisten] connection accepted from 127.0.0.4:37265 #31 (10 connections now open) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31004| Fri Jan 18 13:09:36.416 [conn6] end connection 127.0.0.2:52466 (4 connections now open) m31004| Fri Jan 18 13:09:36.416 [initandlisten] connection accepted from 127.0.0.2:40919 #10 (5 connections now open) m31004| Fri Jan 18 13:09:36.859 [conn7] end connection 127.0.0.5:37054 (4 connections now open) m31004| Fri Jan 18 13:09:36.859 [initandlisten] connection accepted from 127.0.0.5:37773 #11 (5 connections now open) m31004| Fri Jan 18 13:09:37.372 [conn8] end connection 127.0.0.3:59703 (4 connections now open) m31004| Fri Jan 18 13:09:37.372 [initandlisten] connection accepted from 127.0.0.3:60478 #12 (5 connections now open) m31003| Fri Jan 18 13:09:37.421 [conn9] end connection 127.0.0.6:48441 (4 connections now open) m31003| Fri Jan 18 13:09:37.421 [initandlisten] connection accepted from 127.0.0.6:43393 #13 (5 connections now open) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31004| Fri Jan 18 13:09:38.070 [conn9] end connection 127.0.0.4:39569 (4 connections now open) m31004| Fri Jan 18 13:09:38.070 [initandlisten] connection accepted from 127.0.0.4:45345 #13 (5 connections now open) m31002| Fri Jan 18 13:09:38.416 [conn10] end connection 127.0.0.2:54078 (4 connections now open) m31002| Fri Jan 18 13:09:38.416 [initandlisten] connection accepted from 127.0.0.2:57594 #14 (5 connections now open) m31001| Fri Jan 18 13:09:38.859 [conn10] end connection 127.0.0.5:51714 (4 connections now open) m31001| Fri Jan 18 13:09:38.860 [initandlisten] connection accepted from 127.0.0.5:38239 #14 (6 connections now open) m31002| Fri Jan 18 13:09:39.373 [conn11] end connection 127.0.0.3:45916 (4 connections now open) m31002| Fri Jan 18 13:09:39.373 [initandlisten] connection accepted from 127.0.0.3:44584 #15 (5 connections now open) m31001| Fri Jan 18 13:09:39.421 [conn11] end connection 127.0.0.6:33390 (4 connections now open) m31001| Fri Jan 18 13:09:39.421 [initandlisten] connection accepted from 127.0.0.6:54706 #15 (6 connections now open) m31001| Fri Jan 18 13:09:40.070 [conn12] end connection 127.0.0.4:53088 (4 connections now open) m31001| Fri Jan 18 13:09:40.070 [initandlisten] connection accepted from 127.0.0.4:34089 #16 (5 connections now open) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:08:35 GMT-0500 (EST) m31002| Fri Jan 18 13:09:44.412 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31001| Fri Jan 18 13:09:44.412 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:09:44.412 [initandlisten] connection accepted from 127.0.0.4:53044 #32 (11 connections now open) m31000| Fri Jan 18 13:09:44.412 [initandlisten] connection accepted from 127.0.0.3:39611 #33 (12 connections now open) My optime : Fri Jan 18 2013 13:08:35 GMT-0500 (EST) vs Fri Jan 18 2013 13:09:04 GMT-0500 (EST) Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:09:47.444 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dst 127.0.0.5 flowid 1:11 > /tmp/output Fri Jan 18 13:09:47.458 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dst 127.0.0.5 flowid 1:11 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Fri Jan 18 13:09:47.472 shell: started program sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 40000ms > /tmp/output Fri Jan 18 13:09:47.484 shell: started program sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 40000ms > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:09:47.495 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dst 127.0.0.2 flowid 1:12 > /tmp/output Fri Jan 18 13:09:47.509 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dst 127.0.0.2 flowid 1:12 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Fri Jan 18 13:09:47.519 shell: started program sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output Fri Jan 18 13:09:47.529 shell: started program sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output Running: sudo bash -c dig +short undefined > /tmp/output Fri Jan 18 13:09:47.543 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:13 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:09:47.572 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:13 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31001 0xFFFF flowid 1:13 > /tmp/output Fri Jan 18 13:09:47.581 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31001 0xFFFF flowid 1:13 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:13 handle 13: netem > /tmp/output Fri Jan 18 13:09:47.591 shell: started program sudo bash -c tc qdisc add dev lo parent 1:13 handle 13: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:13 handle 13: netem delay 0ms 100ms > /tmp/output Fri Jan 18 13:09:47.605 shell: started program sudo bash -c tc qdisc change dev lo parent 1:13 handle 13: netem delay 0ms 100ms > /tmp/output Running: sudo bash -c dig +short undefined > /tmp/output Fri Jan 18 13:09:47.615 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:14 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:09:47.644 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:14 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31002 0xFFFF flowid 1:14 > /tmp/output Fri Jan 18 13:09:47.656 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31002 0xFFFF flowid 1:14 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:14 handle 14: netem > /tmp/output Fri Jan 18 13:09:47.670 shell: started program sudo bash -c tc qdisc add dev lo parent 1:14 handle 14: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:14 handle 14: netem delay 0ms 100ms > /tmp/output Fri Jan 18 13:09:47.678 shell: started program sudo bash -c tc qdisc change dev lo parent 1:14 handle 14: netem delay 0ms 100ms > /tmp/output Running: sudo bash -c dig +short undefined > /tmp/output Fri Jan 18 13:09:47.692 shell: started program sudo bash -c dig +short undefined > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:15 htb rate 1bps ceil 1000Mbps > /tmp/output Fri Jan 18 13:09:47.716 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:15 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31004 0xFFFF flowid 1:15 > /tmp/output Fri Jan 18 13:09:47.730 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dport 31004 0xFFFF flowid 1:15 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:15 handle 15: netem > /tmp/output Fri Jan 18 13:09:47.744 shell: started program sudo bash -c tc qdisc add dev lo parent 1:15 handle 15: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:15 handle 15: netem delay 500ms > /tmp/output Fri Jan 18 13:09:47.758 shell: started program sudo bash -c tc qdisc change dev lo parent 1:15 handle 15: netem delay 500ms > /tmp/output Running: sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output Fri Jan 18 13:09:47.770 shell: started program sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 4833 ver 3.17 Sent 831351 bytes 4833 pkt (dropped 3, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 11: parent 1:11 limit 1000 delay 40.0s Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 12: parent 1:12 limit 1000 delay 40.0s Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 13: parent 1:13 limit 1000 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 14: parent 1:14 limit 1000 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 15: parent 1:15 limit 1000 delay 500.0ms Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 80, "optime" : { "t" : 1358532590000, "i" : 805 }, "optimeDate" : ISODate("2013-01-18T18:09:50Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:46Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532588000, "i" : 3835 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 45, "optime" : { "t" : 1358532589000, "i" : 3366 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 80, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532589000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 44, "optime" : { "t" : 1358532590000, "i" : 486 }, "optimeDate" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 44, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 44, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 44, "optime" : { "t" : 1358532590000, "i" : 433 }, "optimeDate" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 46, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:49Z"), "pingMs" : 10, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:50Z"), "pingMs" : 58, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 46, "optime" : { "t" : 1358532589000, "i" : 3230 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:49Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 45, "optime" : { "t" : 1358532589000, "i" : 3730 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 45, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358532590000, "i" : 576 }, "optimeDate" : ISODate("2013-01-18T18:09:50Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31002| Fri Jan 18 13:09:52.413 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.ns, filling with zeroes... m31001| Fri Jan 18 13:09:52.413 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.ns, filling with zeroes... m31002| Fri Jan 18 13:09:52.564 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.ns, size: 16MB, took 0.151 secs m31002| Fri Jan 18 13:09:52.564 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.0, filling with zeroes... m31001| Fri Jan 18 13:09:52.581 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.ns, size: 16MB, took 0.167 secs m31001| Fri Jan 18 13:09:52.581 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.0, filling with zeroes... m31002| Fri Jan 18 13:09:52.714 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.0, size: 16MB, took 0.149 secs m31002| Fri Jan 18 13:09:52.716 [repl writer worker 1] build index foo.bar { _id: 1 } m31002| Fri Jan 18 13:09:52.716 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Fri Jan 18 13:09:52.765 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.0, size: 16MB, took 0.183 secs m31001| Fri Jan 18 13:09:52.766 [repl writer worker 1] build index foo.bar { _id: 1 } m31001| Fri Jan 18 13:09:52.766 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31002| Fri Jan 18 13:09:53.020 [conn12] end connection 127.0.0.5:60619 (4 connections now open) m31002| Fri Jan 18 13:09:53.020 [initandlisten] connection accepted from 127.0.0.5:57947 #16 (6 connections now open) m31003| Fri Jan 18 13:09:53.375 [conn11] end connection 127.0.0.3:57511 (4 connections now open) m31003| Fri Jan 18 13:09:53.375 [initandlisten] connection accepted from 127.0.0.3:56970 #14 (5 connections now open) m31002| Fri Jan 18 13:09:53.423 [conn13] end connection 127.0.0.6:60405 (4 connections now open) m31002| Fri Jan 18 13:09:53.423 [initandlisten] connection accepted from 127.0.0.6:47749 #17 (5 connections now open) m31003| Fri Jan 18 13:09:54.072 [conn12] end connection 127.0.0.4:58937 (4 connections now open) m31003| Fri Jan 18 13:09:54.072 [initandlisten] connection accepted from 127.0.0.4:39945 #15 (5 connections now open) m31001| Fri Jan 18 13:09:54.419 [conn13] end connection 127.0.0.2:38379 (4 connections now open) m31001| Fri Jan 18 13:09:54.419 [initandlisten] connection accepted from 127.0.0.2:46091 #17 (5 connections now open) m31000| Fri Jan 18 13:09:55.375 [conn29] end connection 127.0.0.3:34166 (11 connections now open) m31000| Fri Jan 18 13:09:55.375 [initandlisten] connection accepted from 127.0.0.3:32772 #34 (12 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:55Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 85, "optime" : { "t" : 1358532595000, "i" : 3565 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:46Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532594000, "i" : 3010 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 50, "optime" : { "t" : 1358532595000, "i" : 2896 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 85, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532595000, "i" : 2896 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 49, "optime" : { "t" : 1358532594000, "i" : 346 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 84, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358532594000, "i" : 346 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 51, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:55Z"), "pingMs" : 4, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 41, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 51, "optime" : { "t" : 1358532594000, "i" : 2961 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:09:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 50, "optime" : { "t" : 1358532593000, "i" : 2743 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:54Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 50, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:54Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532595000, "i" : 3565 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31000| Fri Jan 18 13:09:55.423 [conn30] end connection 127.0.0.6:44779 (11 connections now open) m31000| Fri Jan 18 13:09:55.424 [initandlisten] connection accepted from 127.0.0.6:40949 #35 (13 connections now open) m31000| Fri Jan 18 13:09:56.073 [conn31] end connection 127.0.0.4:37265 (11 connections now open) m31000| Fri Jan 18 13:09:56.073 [initandlisten] connection accepted from 127.0.0.4:54357 #36 (12 connections now open) m31000| Fri Jan 18 13:09:58.415 [rsHealthPoll] DBClientCursor::init call() failed m31000| Fri Jan 18 13:09:58.415 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31003| Fri Jan 18 13:09:58.859 [rsHealthPoll] DBClientCursor::init call() failed m31003| Fri Jan 18 13:09:58.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Fri Jan 18 13:09:59.415 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Fri Jan 18 13:09:59.415 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31003| Fri Jan 18 13:09:59.859 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Fri Jan 18 13:09:59.859 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 90, "optime" : { "t" : 1358532600000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-18T18:10:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532600000, "i" : 4173 }, "optimeDate" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 55, "optime" : { "t" : 1358532599000, "i" : 3408 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 90, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532599000, "i" : 2628 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 54, "optime" : { "t" : 1358532600000, "i" : 1235 }, "optimeDate" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 54, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 89, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 54, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 54, "optime" : { "t" : 1358532600000, "i" : 1219 }, "optimeDate" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:59Z"), "pingMs" : 22, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 40, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 88, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 56, "optime" : { "t" : 1358532599000, "i" : 2627 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:09:59Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 55, "optime" : { "t" : 1358532599000, "i" : 3813 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532557000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 55, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 88, "optime" : { "t" : 1358532600000, "i" : 6592 }, "optimeDate" : ISODate("2013-01-18T18:10:00Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 95, "optime" : { "t" : 1358532605000, "i" : 8061 }, "optimeDate" : ISODate("2013-01-18T18:10:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532604000, "i" : 3892 }, "optimeDate" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 60, "optime" : { "t" : 1358532605000, "i" : 3774 }, "optimeDate" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 95, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532605000, "i" : 3757 }, "optimeDate" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 59, "optime" : { "t" : 1358532604000, "i" : 565 }, "optimeDate" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 94, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358532603000, "i" : 1364 }, "optimeDate" : ISODate("2013-01-18T18:10:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 25, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 93, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 61, "optime" : { "t" : 1358532604000, "i" : 3138 }, "optimeDate" : ISODate("2013-01-18T18:10:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:05Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 60, "optime" : { "t" : 1358532605000, "i" : 3942 }, "optimeDate" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532562000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 60, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 93, "optime" : { "t" : 1358532605000, "i" : 7622 }, "optimeDate" : ISODate("2013-01-18T18:10:05Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31004| Fri Jan 18 13:10:06.421 [conn10] end connection 127.0.0.2:40919 (4 connections now open) m31004| Fri Jan 18 13:10:06.421 [initandlisten] connection accepted from 127.0.0.2:33396 #14 (5 connections now open) m31004| Fri Jan 18 13:10:07.377 [conn12] end connection 127.0.0.3:60478 (4 connections now open) m31004| Fri Jan 18 13:10:07.377 [initandlisten] connection accepted from 127.0.0.3:40968 #15 (5 connections now open) m31003| Fri Jan 18 13:10:07.425 [conn13] end connection 127.0.0.6:43393 (4 connections now open) m31003| Fri Jan 18 13:10:07.425 [initandlisten] connection accepted from 127.0.0.6:52655 #16 (5 connections now open) m31004| Fri Jan 18 13:10:08.075 [conn13] end connection 127.0.0.4:45345 (4 connections now open) m31004| Fri Jan 18 13:10:08.075 [initandlisten] connection accepted from 127.0.0.4:35320 #16 (5 connections now open) m31002| Fri Jan 18 13:10:08.421 [conn14] end connection 127.0.0.2:57594 (4 connections now open) m31002| Fri Jan 18 13:10:08.421 [initandlisten] connection accepted from 127.0.0.2:37389 #18 (5 connections now open) m31001| Fri Jan 18 13:10:09.051 [conn14] end connection 127.0.0.5:38239 (4 connections now open) m31001| Fri Jan 18 13:10:09.192 [initandlisten] connection accepted from 127.0.0.5:37505 #18 (5 connections now open) m31002| Fri Jan 18 13:10:09.377 [conn15] end connection 127.0.0.3:44584 (4 connections now open) m31002| Fri Jan 18 13:10:09.377 [initandlisten] connection accepted from 127.0.0.3:47848 #19 (5 connections now open) m31001| Fri Jan 18 13:10:09.426 [conn15] end connection 127.0.0.6:54706 (4 connections now open) m31001| Fri Jan 18 13:10:09.426 [initandlisten] connection accepted from 127.0.0.6:38950 #19 (5 connections now open) m31001| Fri Jan 18 13:10:10.075 [conn16] end connection 127.0.0.4:34089 (4 connections now open) m31001| Fri Jan 18 13:10:10.075 [initandlisten] connection accepted from 127.0.0.4:36273 #20 (5 connections now open) m31000| Fri Jan 18 13:10:10.669 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.1, filling with zeroes... m31000| Fri Jan 18 13:10:10.946 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.1, size: 32MB, took 0.277 secs m31000| Fri Jan 18 13:10:10.946 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:277941 277ms m31004| Fri Jan 18 13:10:10.947 [FileAllocator] allocating new datafile /data/db/testReplSet-4/foo.1, filling with zeroes... STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 101, "optime" : { "t" : 1358532611000, "i" : 4341 }, "optimeDate" : ISODate("2013-01-18T18:10:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532610000, "i" : 3507 }, "optimeDate" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 66, "optime" : { "t" : 1358532609000, "i" : 3968 }, "optimeDate" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 101, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:11Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532609000, "i" : 3959 }, "optimeDate" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 65, "optime" : { "t" : 1358532610000, "i" : 330 }, "optimeDate" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 65, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 100, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 65, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 65, "optime" : { "t" : 1358532610000, "i" : 302 }, "optimeDate" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 24, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 20, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 99, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 67, "optime" : { "t" : 1358532609000, "i" : 3851 }, "optimeDate" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:09Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 66, "optime" : { "t" : 1358532609000, "i" : 4208 }, "optimeDate" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532567000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:10Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 66, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 99, "optime" : { "t" : 1358532610000, "i" : 5337 }, "optimeDate" : ISODate("2013-01-18T18:10:10Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31004| Fri Jan 18 13:10:11.275 [FileAllocator] done allocating datafile /data/db/testReplSet-4/foo.1, size: 32MB, took 0.328 secs m31000| Fri Jan 18 13:10:11.415 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31003| Fri Jan 18 13:10:11.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31004| Fri Jan 18 13:10:11.864 [conn11] end connection 127.0.0.5:37773 (4 connections now open) m31004| Fri Jan 18 13:10:12.364 [initandlisten] connection accepted from 127.0.0.5:41535 #17 (5 connections now open) m31000| Fri Jan 18 13:10:14.765 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:297667 148ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 106, "optime" : { "t" : 1358532616000, "i" : 3423 }, "optimeDate" : ISODate("2013-01-18T18:10:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:16Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532614000, "i" : 1807 }, "optimeDate" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 71, "optime" : { "t" : 1358532615000, "i" : 3049 }, "optimeDate" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 106, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:16Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532615000, "i" : 3049 }, "optimeDate" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 70, "optime" : { "t" : 1358532616000, "i" : 986 }, "optimeDate" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 70, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 105, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 70, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 70, "optime" : { "t" : 1358532616000, "i" : 973 }, "optimeDate" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:15Z"), "pingMs" : 30, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:16Z"), "pingMs" : 31, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 104, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 72, "optime" : { "t" : 1358532614000, "i" : 6040 }, "optimeDate" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:15Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 71, "optime" : { "t" : 1358532615000, "i" : 3730 }, "optimeDate" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532572000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:16Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 71, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 104, "optime" : { "t" : 1358532616000, "i" : 2725 }, "optimeDate" : ISODate("2013-01-18T18:10:16Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31003| Fri Jan 18 13:10:17.439 [rsBackgroundSync] Socket recv() timeout 127.0.0.2:31000 m31003| Fri Jan 18 13:10:17.439 [rsBackgroundSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [3] server [127.0.0.2:31000] m31003| Fri Jan 18 13:10:17.439 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Fri Jan 18 13:10:19.899 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:644254 322ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 111, "optime" : { "t" : 1358532621000, "i" : 4965 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532620000, "i" : 4819 }, "optimeDate" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 76, "optime" : { "t" : 1358532621000, "i" : 3819 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532621000, "i" : 2872 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 75, "optime" : { "t" : 1358532620000, "i" : 393 }, "optimeDate" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 75, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 110, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 75, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 75, "optime" : { "t" : 1358532619000, "i" : 4681 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 24, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 33, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 109, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358532619000, "i" : 1781 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:21Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:21Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 76, "optime" : { "t" : 1358532621000, "i" : 4195 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532577000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 109, "optime" : { "t" : 1358532621000, "i" : 4817 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31000| Fri Jan 18 13:10:22.840 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 locks(micros) r:119663 nreturned:140001 reslen:2380037 119ms m31000| Fri Jan 18 13:10:22.881 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 locks(micros) r:110223 nreturned:140001 reslen:2380037 110ms m31003| Fri Jan 18 13:10:23.379 [conn14] end connection 127.0.0.3:56970 (4 connections now open) m31003| Fri Jan 18 13:10:23.380 [initandlisten] connection accepted from 127.0.0.3:45637 #17 (6 connections now open) m31002| Fri Jan 18 13:10:23.428 [conn17] end connection 127.0.0.6:47749 (4 connections now open) m31002| Fri Jan 18 13:10:23.428 [initandlisten] connection accepted from 127.0.0.6:44708 #20 (5 connections now open) m31002| Fri Jan 18 13:10:23.511 [conn16] end connection 127.0.0.5:57947 (4 connections now open) m31002| Fri Jan 18 13:10:23.511 [initandlisten] connection accepted from 127.0.0.5:40963 #21 (6 connections now open) m31003| Fri Jan 18 13:10:24.077 [conn15] end connection 127.0.0.4:39945 (4 connections now open) m31003| Fri Jan 18 13:10:24.077 [initandlisten] connection accepted from 127.0.0.4:44589 #18 (5 connections now open) m31000| Fri Jan 18 13:10:24.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Fri Jan 18 13:10:24.424 [conn17] end connection 127.0.0.2:46091 (4 connections now open) m31001| Fri Jan 18 13:10:24.424 [initandlisten] connection accepted from 127.0.0.2:44186 #21 (5 connections now open) m31003| Fri Jan 18 13:10:24.859 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Fri Jan 18 13:10:25.380 [conn34] end connection 127.0.0.3:32772 (11 connections now open) m31000| Fri Jan 18 13:10:25.380 [initandlisten] connection accepted from 127.0.0.3:45985 #37 (13 connections now open) m31000| Fri Jan 18 13:10:25.428 [conn35] end connection 127.0.0.6:40949 (11 connections now open) m31000| Fri Jan 18 13:10:25.428 [initandlisten] connection accepted from 127.0.0.6:35422 #38 (12 connections now open) m31000| Fri Jan 18 13:10:25.707 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:2253641 1126ms m31000| Fri Jan 18 13:10:26.077 [conn36] end connection 127.0.0.4:54357 (11 connections now open) m31000| Fri Jan 18 13:10:26.077 [initandlisten] connection accepted from 127.0.0.4:35856 #39 (12 connections now open) m31003| Fri Jan 18 13:10:26.413 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 117, "optime" : { "t" : 1358532627000, "i" : 1596 }, "optimeDate" : ISODate("2013-01-18T18:10:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532626000, "i" : 6881 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 82, "optime" : { "t" : 1358532624000, "i" : 360 }, "optimeDate" : ISODate("2013-01-18T18:10:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 82, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:26Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 82, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 82, "optime" : { "t" : 1358532624000, "i" : 349 }, "optimeDate" : ISODate("2013-01-18T18:10:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 81, "optime" : { "t" : 1358532626000, "i" : 2407 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358532626000, "i" : 825 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 m31000| Fri Jan 18 13:10:34.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31004| Fri Jan 18 13:10:36.426 [conn14] end connection 127.0.0.2:33396 (4 connections now open) m31004| Fri Jan 18 13:10:36.426 [initandlisten] connection accepted from 127.0.0.2:54375 #18 (5 connections now open) m31004| Fri Jan 18 13:10:37.382 [conn15] end connection 127.0.0.3:40968 (4 connections now open) m31004| Fri Jan 18 13:10:37.382 [initandlisten] connection accepted from 127.0.0.3:49636 #19 (5 connections now open) m31003| Fri Jan 18 13:10:37.430 [conn16] end connection 127.0.0.6:52655 (4 connections now open) m31003| Fri Jan 18 13:10:37.430 [initandlisten] connection accepted from 127.0.0.6:37066 #19 (5 connections now open) m31003| Fri Jan 18 13:10:37.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31004| Fri Jan 18 13:10:38.079 [conn16] end connection 127.0.0.4:35320 (4 connections now open) m31004| Fri Jan 18 13:10:38.079 [initandlisten] connection accepted from 127.0.0.4:45556 #20 (5 connections now open) m31003| Fri Jan 18 13:10:38.415 [conn10] end connection 127.0.0.2:60302 (4 connections now open) m31002| Fri Jan 18 13:10:38.426 [conn18] end connection 127.0.0.2:37389 (4 connections now open) m31002| Fri Jan 18 13:10:38.426 [initandlisten] connection accepted from 127.0.0.2:57262 #22 (5 connections now open) m31000| Fri Jan 18 13:10:38.859 [conn28] end connection 127.0.0.5:35703 (11 connections now open) m31002| Fri Jan 18 13:10:39.382 [conn19] end connection 127.0.0.3:47848 (4 connections now open) m31002| Fri Jan 18 13:10:39.382 [initandlisten] connection accepted from 127.0.0.3:38330 #23 (5 connections now open) m31001| Fri Jan 18 13:10:39.430 [conn19] end connection 127.0.0.6:38950 (4 connections now open) m31001| Fri Jan 18 13:10:39.430 [initandlisten] connection accepted from 127.0.0.6:50946 #22 (5 connections now open) m31001| Fri Jan 18 13:10:39.726 [conn18] end connection 127.0.0.5:37505 (4 connections now open) m31001| Fri Jan 18 13:10:39.763 [initandlisten] connection accepted from 127.0.0.5:39483 #23 (5 connections now open) m31001| Fri Jan 18 13:10:40.079 [conn20] end connection 127.0.0.4:36273 (4 connections now open) m31001| Fri Jan 18 13:10:40.080 [initandlisten] connection accepted from 127.0.0.4:48258 #24 (5 connections now open) m31003| Fri Jan 18 13:10:47.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31004| Fri Jan 18 13:10:49.869 [conn17] end connection 127.0.0.5:41535 (4 connections now open) m31004| Fri Jan 18 13:10:50.369 [initandlisten] connection accepted from 127.0.0.5:42308 #21 (5 connections now open) m31001| Fri Jan 18 13:10:51.633 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.1, filling with zeroes... m31002| Fri Jan 18 13:10:51.748 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.1, filling with zeroes... m31001| Fri Jan 18 13:10:51.913 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.1, size: 32MB, took 0.28 secs m31002| Fri Jan 18 13:10:52.022 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.1, size: 32MB, took 0.274 secs m31003| Fri Jan 18 13:10:53.385 [conn17] end connection 127.0.0.3:45637 (3 connections now open) m31003| Fri Jan 18 13:10:53.385 [initandlisten] connection accepted from 127.0.0.3:50806 #20 (5 connections now open) m31002| Fri Jan 18 13:10:53.433 [conn20] end connection 127.0.0.6:44708 (4 connections now open) m31002| Fri Jan 18 13:10:53.433 [initandlisten] connection accepted from 127.0.0.6:49816 #24 (5 connections now open) m31002| Fri Jan 18 13:10:53.793 [conn21] end connection 127.0.0.5:40963 (4 connections now open) m31002| Fri Jan 18 13:10:53.850 [initandlisten] connection accepted from 127.0.0.5:54904 #25 (5 connections now open) m31003| Fri Jan 18 13:10:54.082 [conn18] end connection 127.0.0.4:44589 (3 connections now open) m31003| Fri Jan 18 13:10:54.082 [initandlisten] connection accepted from 127.0.0.4:36155 #21 (4 connections now open) m31001| Fri Jan 18 13:10:54.429 [conn21] end connection 127.0.0.2:44186 (4 connections now open) m31001| Fri Jan 18 13:10:54.429 [initandlisten] connection accepted from 127.0.0.2:51931 #25 (6 connections now open) m31000| Fri Jan 18 13:10:55.385 [conn37] end connection 127.0.0.3:45985 (10 connections now open) m31000| Fri Jan 18 13:10:55.385 [initandlisten] connection accepted from 127.0.0.3:43472 #40 (11 connections now open) m31000| Fri Jan 18 13:10:55.433 [conn38] end connection 127.0.0.6:35422 (10 connections now open) m31000| Fri Jan 18 13:10:55.433 [initandlisten] connection accepted from 127.0.0.6:42741 #41 (11 connections now open) m31000| Fri Jan 18 13:10:56.082 [conn39] end connection 127.0.0.4:35856 (10 connections now open) m31000| Fri Jan 18 13:10:56.082 [initandlisten] connection accepted from 127.0.0.4:60453 #42 (12 connections now open) m31003| Fri Jan 18 13:10:56.411 [rsSyncNotifier] repl: couldn't connect to server 127.0.0.2:31000 m31003| Fri Jan 18 13:10:56.411 [conn1] command admin.$cmd command: { replSetGetStatus: true } ntoreturn:1 keyUpdates:0 reslen:1373 29229ms { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 25, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:26Z"), "pingMs" : 20, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 115, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : { "t" : 1358532624000, "i" : 349 }, "optimeDate" : ISODate("2013-01-18T18:10:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:25Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31000| Fri Jan 18 13:10:56.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:10:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 111, "optime" : { "t" : 1358532627000, "i" : 1596 }, "optimeDate" : ISODate("2013-01-18T18:10:27Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358532615000, "i" : 2308 }, "optimeDate" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358532615000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:10:56Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 111, "optime" : { "t" : 1358532544000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 144, "optime" : { "t" : 1358532627000, "i" : 1596 }, "optimeDate" : ISODate("2013-01-18T18:10:27Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31003| Fri Jan 18 13:10:56.448 [FileAllocator] allocating new datafile /data/db/testReplSet-3/foo.ns, filling with zeroes... m31003| Fri Jan 18 13:10:56.597 [FileAllocator] done allocating datafile /data/db/testReplSet-3/foo.ns, size: 16MB, took 0.149 secs m31003| Fri Jan 18 13:10:56.597 [FileAllocator] allocating new datafile /data/db/testReplSet-3/foo.0, filling with zeroes... m31003| Fri Jan 18 13:10:56.764 [FileAllocator] done allocating datafile /data/db/testReplSet-3/foo.0, size: 16MB, took 0.166 secs m31003| Fri Jan 18 13:10:56.765 [repl writer worker 1] build index foo.bar { _id: 1 } m31003| Fri Jan 18 13:10:56.766 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31003| Fri Jan 18 13:10:57.411 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Fri Jan 18 13:10:57.439 [conn24] end connection 127.0.0.5:45311 (10 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 151, "optime" : { "t" : 1358532661000, "i" : 9861 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532619000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532619000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532660000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 116, "optime" : { "t" : 1358532661000, "i" : 5333 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 151, "optime" : { "t" : 1358532621000, "i" : 4650 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358532620000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 116, "optime" : { "t" : 1358532661000, "i" : 5295 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 115, "optime" : { "t" : 1358532660000, "i" : 1139 }, "optimeDate" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 115, "optime" : { "t" : 1358532619000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 150, "optime" : { "t" : 1358532621000, "i" : 4650 }, "optimeDate" : ISODate("2013-01-18T18:10:21Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 115, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 115, "optime" : { "t" : 1358532660000, "i" : 1129 }, "optimeDate" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 m31000| Fri Jan 18 13:11:06.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31004| Fri Jan 18 13:11:06.431 [conn18] end connection 127.0.0.2:54375 (4 connections now open) m31004| Fri Jan 18 13:11:06.431 [initandlisten] connection accepted from 127.0.0.2:37340 #22 (6 connections now open) m31004| Fri Jan 18 13:11:07.387 [conn19] end connection 127.0.0.3:49636 (4 connections now open) m31004| Fri Jan 18 13:11:07.387 [initandlisten] connection accepted from 127.0.0.3:42430 #23 (5 connections now open) m31003| Fri Jan 18 13:11:07.435 [conn19] end connection 127.0.0.6:37066 (3 connections now open) m31003| Fri Jan 18 13:11:07.435 [initandlisten] connection accepted from 127.0.0.6:45510 #22 (5 connections now open) m31004| Fri Jan 18 13:11:08.084 [conn20] end connection 127.0.0.4:45556 (4 connections now open) m31004| Fri Jan 18 13:11:08.084 [initandlisten] connection accepted from 127.0.0.4:40990 #24 (5 connections now open) m31002| Fri Jan 18 13:11:08.432 [conn22] end connection 127.0.0.2:57262 (4 connections now open) m31002| Fri Jan 18 13:11:08.432 [initandlisten] connection accepted from 127.0.0.2:40874 #26 (5 connections now open) m31002| Fri Jan 18 13:11:09.387 [conn23] end connection 127.0.0.3:38330 (4 connections now open) m31002| Fri Jan 18 13:11:09.387 [initandlisten] connection accepted from 127.0.0.3:46440 #27 (6 connections now open) m31001| Fri Jan 18 13:11:09.435 [conn22] end connection 127.0.0.6:50946 (4 connections now open) m31001| Fri Jan 18 13:11:09.435 [initandlisten] connection accepted from 127.0.0.6:60540 #26 (5 connections now open) m31003| Fri Jan 18 13:11:09.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Fri Jan 18 13:11:10.085 [conn24] end connection 127.0.0.4:48258 (4 connections now open) m31001| Fri Jan 18 13:11:10.085 [initandlisten] connection accepted from 127.0.0.4:52753 #27 (5 connections now open) m31003| Fri Jan 18 13:11:10.152 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Fri Jan 18 13:11:10.180 [conn23] end connection 127.0.0.5:39483 (4 connections now open) m31001| Fri Jan 18 13:11:10.336 [initandlisten] connection accepted from 127.0.0.5:36771 #28 (5 connections now open) m31003| Fri Jan 18 13:11:19.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Fri Jan 18 13:11:23.390 [conn20] end connection 127.0.0.3:50806 (3 connections now open) m31003| Fri Jan 18 13:11:23.390 [initandlisten] connection accepted from 127.0.0.3:40368 #23 (5 connections now open) m31002| Fri Jan 18 13:11:23.438 [conn24] end connection 127.0.0.6:49816 (4 connections now open) m31002| Fri Jan 18 13:11:23.438 [initandlisten] connection accepted from 127.0.0.6:38590 #28 (5 connections now open) m31003| Fri Jan 18 13:11:24.087 [conn21] end connection 127.0.0.4:36155 (3 connections now open) m31003| Fri Jan 18 13:11:24.088 [initandlisten] connection accepted from 127.0.0.4:52017 #24 (4 connections now open) m31002| Fri Jan 18 13:11:24.387 [conn25] end connection 127.0.0.5:54904 (4 connections now open) m31002| Fri Jan 18 13:11:24.413 [initandlisten] connection accepted from 127.0.0.5:37355 #29 (5 connections now open) m31001| Fri Jan 18 13:11:24.435 [conn25] end connection 127.0.0.2:51931 (4 connections now open) m31001| Fri Jan 18 13:11:24.435 [initandlisten] connection accepted from 127.0.0.2:36052 #29 (5 connections now open) m31000| Fri Jan 18 13:11:25.390 [conn40] end connection 127.0.0.3:43472 (9 connections now open) m31000| Fri Jan 18 13:11:25.390 [initandlisten] connection accepted from 127.0.0.3:41671 #43 (10 connections now open) m31000| Fri Jan 18 13:11:25.438 [conn41] end connection 127.0.0.6:42741 (9 connections now open) m31000| Fri Jan 18 13:11:25.438 [initandlisten] connection accepted from 127.0.0.6:38776 #44 (10 connections now open) m31000| Fri Jan 18 13:11:26.088 [conn42] end connection 127.0.0.4:60453 (9 connections now open) m31000| Fri Jan 18 13:11:26.088 [initandlisten] connection accepted from 127.0.0.4:35855 #45 (10 connections now open) m31003| Fri Jan 18 13:11:27.411 [rsSyncNotifier] repl: couldn't connect to server 127.0.0.2:31000 m31003| Fri Jan 18 13:11:27.411 [conn1] command admin.$cmd command: { replSetGetStatus: true } ntoreturn:1 keyUpdates:0 reslen:1373 25755ms { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532619000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 35, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532619000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:10:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:00Z"), "pingMs" : 25, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 149, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358532660000, "i" : 2786 }, "optimeDate" : ISODate("2013-01-18T18:11:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:10:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:01Z"), "pingMs" : 500, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 142, "optime" : { "t" : 1358532661000, "i" : 9861 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:27Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:26Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358532552000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532661000, "i" : 9861 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.5:31003", "127.0.0.6:31004" ] } } m31003| Fri Jan 18 13:11:27.452 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:27.452 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:27.452 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:27.452 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:11:27.452 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:11:27.453 [initandlisten] connection accepted from 127.0.0.5:44820 #30 (6 connections now open) m31001| Fri Jan 18 13:11:27.539 [conn30] end connection 127.0.0.5:44820 (5 connections now open) m31003| Fri Jan 18 13:11:27.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:27.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:27.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:27.543 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:27.620 [initandlisten] connection accepted from 127.0.0.5:56042 #31 (6 connections now open) m31003| Fri Jan 18 13:11:27.842 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:27.842 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:27.842 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:27.842 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31004| Fri Jan 18 13:11:27.875 [conn21] end connection 127.0.0.5:42308 (4 connections now open) m31001| Fri Jan 18 13:11:27.918 [conn31] end connection 127.0.0.5:56042 (5 connections now open) m31001| Fri Jan 18 13:11:27.918 [initandlisten] connection accepted from 127.0.0.5:47993 #32 (7 connections now open) m31003| Fri Jan 18 13:11:28.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:28.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:28.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:28.009 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.015 [conn32] end connection 127.0.0.5:47993 (5 connections now open) m31001| Fri Jan 18 13:11:28.097 [initandlisten] connection accepted from 127.0.0.5:40251 #33 (6 connections now open) m31001| Fri Jan 18 13:11:28.209 [conn33] end connection 127.0.0.5:40251 (5 connections now open) m31003| Fri Jan 18 13:11:28.213 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:28.213 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:28.213 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:28.213 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.250 [initandlisten] connection accepted from 127.0.0.5:48348 #34 (6 connections now open) m31004| Fri Jan 18 13:11:28.375 [initandlisten] connection accepted from 127.0.0.5:44848 #25 (5 connections now open) m31003| Fri Jan 18 13:11:28.412 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.412 [initandlisten] connection accepted from 127.0.0.5:38271 #35 (7 connections now open) m31000| Fri Jan 18 13:11:28.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Fri Jan 18 13:11:28.467 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:28.467 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:28.467 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:28.467 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.542 [initandlisten] connection accepted from 127.0.0.5:40807 #36 (8 connections now open) m31001| Fri Jan 18 13:11:28.542 [conn34] end connection 127.0.0.5:48348 (7 connections now open) m31001| Fri Jan 18 13:11:28.589 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Fri Jan 18 13:11:28.589 [initandlisten] connection accepted from 127.0.0.3:51322 #46 (11 connections now open) m31003| Fri Jan 18 13:11:28.604 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:28.604 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:28.604 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:28.604 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.610 [conn36] end connection 127.0.0.5:40807 (6 connections now open) m31001| Fri Jan 18 13:11:28.610 [initandlisten] connection accepted from 127.0.0.5:43112 #37 (8 connections now open) m31001| Fri Jan 18 13:11:28.717 [conn35] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532587000|401 } } cursorid:753746169473790 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:131315 nreturned:156051 reslen:2652887 128ms m31003| Fri Jan 18 13:11:28.802 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:28.802 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:28.802 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:28.802 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:28.850 [conn37] end connection 127.0.0.5:43112 (6 connections now open) m31001| Fri Jan 18 13:11:28.950 [initandlisten] connection accepted from 127.0.0.5:50143 #38 (7 connections now open) m31003| Fri Jan 18 13:11:29.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:29.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:29.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:29.037 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:29.040 [conn38] end connection 127.0.0.5:50143 (6 connections now open) m31001| Fri Jan 18 13:11:29.122 [initandlisten] connection accepted from 127.0.0.5:53705 #39 (7 connections now open) m31001| Fri Jan 18 13:11:29.237 [conn39] end connection 127.0.0.5:53705 (6 connections now open) m31003| Fri Jan 18 13:11:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:29.241 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:29.241 [initandlisten] connection accepted from 127.0.0.5:48151 #40 (7 connections now open) m31003| Fri Jan 18 13:11:29.380 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:29.380 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:29.380 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:29.381 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:29.420 [conn40] end connection 127.0.0.5:48151 (6 connections now open) m31001| Fri Jan 18 13:11:29.434 [initandlisten] connection accepted from 127.0.0.5:48906 #41 (7 connections now open) m31003| Fri Jan 18 13:11:29.555 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:29.555 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:29.555 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:29.555 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:29.590 [slaveTracking] build index local.slaves { _id: 1 } m31001| Fri Jan 18 13:11:29.590 [slaveTracking] build index done. scanned 0 total records. 0 secs m31001| Fri Jan 18 13:11:29.607 [conn41] end connection 127.0.0.5:48906 (6 connections now open) m31001| Fri Jan 18 13:11:29.608 [initandlisten] connection accepted from 127.0.0.5:44082 #42 (8 connections now open) m31003| Fri Jan 18 13:11:29.833 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:29.833 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:29.833 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:29.833 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:29.878 [conn42] end connection 127.0.0.5:44082 (6 connections now open) m31001| Fri Jan 18 13:11:29.958 [initandlisten] connection accepted from 127.0.0.5:41819 #43 (7 connections now open) m31001| Fri Jan 18 13:11:30.007 [conn43] end connection 127.0.0.5:41819 (6 connections now open) m31003| Fri Jan 18 13:11:30.011 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:30.011 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:30.011 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:30.011 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:30.156 [initandlisten] connection accepted from 127.0.0.5:45482 #44 (7 connections now open) m31003| Fri Jan 18 13:11:30.295 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:30.296 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:30.296 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:30.296 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:30.389 [initandlisten] connection accepted from 127.0.0.5:50137 #45 (8 connections now open) m31001| Fri Jan 18 13:11:30.389 [conn44] end connection 127.0.0.5:45482 (6 connections now open) m31001| Fri Jan 18 13:11:30.515 [conn45] end connection 127.0.0.5:50137 (6 connections now open) m31000| Fri Jan 18 13:11:30.711 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:420965 210ms m31003| Fri Jan 18 13:11:30.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:30.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:30.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:30.714 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:30.729 [initandlisten] connection accepted from 127.0.0.5:46848 #46 (7 connections now open) m31003| Fri Jan 18 13:11:30.866 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:30.866 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:30.866 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:30.866 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:30.916 [initandlisten] connection accepted from 127.0.0.5:54373 #47 (8 connections now open) m31001| Fri Jan 18 13:11:30.916 [conn46] end connection 127.0.0.5:46848 (7 connections now open) m31001| Fri Jan 18 13:11:31.126 [conn47] end connection 127.0.0.5:54373 (6 connections now open) m31003| Fri Jan 18 13:11:31.129 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:31.129 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:31.129 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:31.129 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31000| Fri Jan 18 13:11:31.153 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:342716 171ms m31001| Fri Jan 18 13:11:31.240 [initandlisten] connection accepted from 127.0.0.5:35409 #48 (7 connections now open) m31001| Fri Jan 18 13:11:31.287 [conn48] end connection 127.0.0.5:35409 (6 connections now open) m31003| Fri Jan 18 13:11:31.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:31.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:31.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:31.290 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:31.290 [initandlisten] connection accepted from 127.0.0.5:46795 #49 (7 connections now open) m31003| Fri Jan 18 13:11:31.427 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:31.427 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:31.427 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:31.427 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:31.449 [conn49] end connection 127.0.0.5:46795 (6 connections now open) m31001| Fri Jan 18 13:11:31.449 [initandlisten] connection accepted from 127.0.0.5:34327 #50 (8 connections now open) m31001| Fri Jan 18 13:11:31.719 [conn50] end connection 127.0.0.5:34327 (6 connections now open) m31003| Fri Jan 18 13:11:31.721 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:31.721 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:31.721 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:31.721 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:31.846 [initandlisten] connection accepted from 127.0.0.5:43497 #51 (7 connections now open) m31003| Fri Jan 18 13:11:31.967 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:31.967 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:31.967 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:31.967 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:31.992 [conn51] end connection 127.0.0.5:43497 (6 connections now open) m31001| Fri Jan 18 13:11:32.044 [initandlisten] connection accepted from 127.0.0.5:47460 #52 (7 connections now open) m31003| Fri Jan 18 13:11:32.108 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.108 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:32.108 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:32.108 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:32.183 [conn52] end connection 127.0.0.5:47460 (6 connections now open) m31001| Fri Jan 18 13:11:32.191 [initandlisten] connection accepted from 127.0.0.5:44539 #53 (7 connections now open) m31003| Fri Jan 18 13:11:32.319 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.319 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:32.319 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:32.319 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:32.354 [initandlisten] connection accepted from 127.0.0.5:56194 #54 (8 connections now open) m31001| Fri Jan 18 13:11:32.354 [conn53] end connection 127.0.0.5:44539 (7 connections now open) m31003| Fri Jan 18 13:11:32.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:32.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:32.572 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 STATUS FROM : 127.0.0.2:31000 m31001| Fri Jan 18 13:11:32.600 [conn54] end connection 127.0.0.5:56194 (6 connections now open) m31001| Fri Jan 18 13:11:32.606 [initandlisten] connection accepted from 127.0.0.5:49918 #55 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 182, "optime" : { "t" : 1358532692000, "i" : 5258 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532692000, "i" : 3170 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 147, "optime" : { "t" : 1358532691000, "i" : 3635 }, "optimeDate" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 182, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532588000, "i" : 341 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532691000, "i" : 3628 }, "optimeDate" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 146, "optime" : { "t" : 1358532692000, "i" : 92 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 181, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358532588000, "i" : 641 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358532692000, "i" : 83 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 m31003| Fri Jan 18 13:11:32.743 [conn1] command admin.$cmd command: { replSetGetStatus: true } ntoreturn:1 keyUpdates:0 reslen:1149 119ms { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:32Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 11 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 34 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532588000, "i" : 941 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358532690000, "i" : 3612 }, "optimeDate" : ISODate("2013-01-18T18:11:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:31Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.747 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.747 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:32.747 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:32.747 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 147, "optime" : { "t" : 1358532691000, "i" : 3746 }, "optimeDate" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:32Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358532588000, "i" : 441 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532692000, "i" : 5258 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:11:32.771 [conn55] end connection 127.0.0.5:49918 (6 connections now open) m31001| Fri Jan 18 13:11:32.844 [initandlisten] connection accepted from 127.0.0.5:39920 #56 (7 connections now open) m31003| Fri Jan 18 13:11:32.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:32.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:32.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:32.961 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:33.051 [conn56] end connection 127.0.0.5:39920 (6 connections now open) m31001| Fri Jan 18 13:11:33.051 [initandlisten] connection accepted from 127.0.0.5:35649 #57 (8 connections now open) m31003| Fri Jan 18 13:11:33.286 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:33.286 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:33.286 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:33.286 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:33.327 [conn57] end connection 127.0.0.5:35649 (6 connections now open) m31001| Fri Jan 18 13:11:33.327 [initandlisten] connection accepted from 127.0.0.5:41173 #58 (8 connections now open) m31003| Fri Jan 18 13:11:33.525 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:33.525 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:33.525 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:33.525 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:33.552 [conn58] end connection 127.0.0.5:41173 (6 connections now open) m31001| Fri Jan 18 13:11:33.604 [initandlisten] connection accepted from 127.0.0.5:45929 #59 (7 connections now open) m31003| Fri Jan 18 13:11:33.718 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:33.718 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:33.718 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:33.718 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:33.806 [conn59] end connection 127.0.0.5:45929 (6 connections now open) m31001| Fri Jan 18 13:11:33.818 [initandlisten] connection accepted from 127.0.0.5:42749 #60 (7 connections now open) m31001| Fri Jan 18 13:11:33.861 [conn60] end connection 127.0.0.5:42749 (6 connections now open) m31003| Fri Jan 18 13:11:33.865 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:33.865 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:33.865 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:33.865 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:33.897 [initandlisten] connection accepted from 127.0.0.5:40048 #61 (7 connections now open) m31003| Fri Jan 18 13:11:33.983 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:33.983 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:33.983 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:33.983 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:34.053 [conn61] end connection 127.0.0.5:40048 (6 connections now open) m31001| Fri Jan 18 13:11:34.100 [initandlisten] connection accepted from 127.0.0.5:36888 #62 (7 connections now open) m31003| Fri Jan 18 13:11:34.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:34.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:34.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:34.222 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:34.262 [conn62] end connection 127.0.0.5:36888 (6 connections now open) m31001| Fri Jan 18 13:11:34.305 [initandlisten] connection accepted from 127.0.0.5:42682 #63 (7 connections now open) m31003| Fri Jan 18 13:11:34.357 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:34.357 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:34.357 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:34.357 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:34.436 [conn63] end connection 127.0.0.5:42682 (6 connections now open) m31001| Fri Jan 18 13:11:34.524 [initandlisten] connection accepted from 127.0.0.5:38033 #64 (7 connections now open) m31003| Fri Jan 18 13:11:34.578 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:34.578 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:34.578 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:34.578 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:34.639 [initandlisten] connection accepted from 127.0.0.5:48349 #65 (8 connections now open) m31001| Fri Jan 18 13:11:34.639 [conn64] end connection 127.0.0.5:38033 (6 connections now open) m31001| Fri Jan 18 13:11:34.735 [conn65] end connection 127.0.0.5:48349 (6 connections now open) m31003| Fri Jan 18 13:11:34.739 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:34.739 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:34.739 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:34.739 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:34.782 [initandlisten] connection accepted from 127.0.0.5:48326 #66 (7 connections now open) m31001| Fri Jan 18 13:11:34.892 [conn66] end connection 127.0.0.5:48326 (6 connections now open) m31003| Fri Jan 18 13:11:34.896 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:34.896 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:34.896 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:34.896 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.008 [initandlisten] connection accepted from 127.0.0.5:37459 #67 (7 connections now open) m31003| Fri Jan 18 13:11:35.062 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.062 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.062 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.062 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.129 [conn67] end connection 127.0.0.5:37459 (6 connections now open) m31001| Fri Jan 18 13:11:35.129 [initandlisten] connection accepted from 127.0.0.5:60992 #68 (8 connections now open) m31003| Fri Jan 18 13:11:35.180 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.180 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.180 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.180 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.224 [conn68] end connection 127.0.0.5:60992 (6 connections now open) m31001| Fri Jan 18 13:11:35.244 [initandlisten] connection accepted from 127.0.0.5:49779 #69 (7 connections now open) m31001| Fri Jan 18 13:11:35.358 [conn69] end connection 127.0.0.5:49779 (6 connections now open) m31003| Fri Jan 18 13:11:35.394 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.394 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.394 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.394 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.420 [initandlisten] connection accepted from 127.0.0.5:55607 #70 (7 connections now open) m31003| Fri Jan 18 13:11:35.566 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.566 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.566 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.566 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.589 [conn70] end connection 127.0.0.5:55607 (6 connections now open) m31001| Fri Jan 18 13:11:35.637 [initandlisten] connection accepted from 127.0.0.5:47882 #71 (7 connections now open) m31003| Fri Jan 18 13:11:35.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.784 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.858 [conn71] end connection 127.0.0.5:47882 (6 connections now open) m31001| Fri Jan 18 13:11:35.858 [initandlisten] connection accepted from 127.0.0.5:46280 #72 (8 connections now open) m31003| Fri Jan 18 13:11:35.957 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:35.957 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:35.957 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:35.957 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:35.973 [conn72] end connection 127.0.0.5:46280 (6 connections now open) m31001| Fri Jan 18 13:11:36.069 [initandlisten] connection accepted from 127.0.0.5:53808 #73 (7 connections now open) m31003| Fri Jan 18 13:11:36.187 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:36.187 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:36.187 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:36.187 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:36.246 [initandlisten] connection accepted from 127.0.0.5:34049 #74 (8 connections now open) m31001| Fri Jan 18 13:11:36.246 [conn73] end connection 127.0.0.5:53808 (7 connections now open) m31001| Fri Jan 18 13:11:36.370 [conn74] end connection 127.0.0.5:34049 (6 connections now open) m31003| Fri Jan 18 13:11:36.372 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:36.372 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:36.372 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:36.372 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:36.436 [initandlisten] connection accepted from 127.0.0.5:34188 #75 (7 connections now open) m31004| Fri Jan 18 13:11:36.437 [conn22] end connection 127.0.0.2:37340 (4 connections now open) m31004| Fri Jan 18 13:11:36.437 [initandlisten] connection accepted from 127.0.0.2:33659 #26 (5 connections now open) m31003| Fri Jan 18 13:11:36.545 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:36.545 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:36.545 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:36.545 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:36.598 [initandlisten] connection accepted from 127.0.0.5:45911 #76 (8 connections now open) m31001| Fri Jan 18 13:11:36.598 [conn75] end connection 127.0.0.5:34188 (7 connections now open) m31003| Fri Jan 18 13:11:36.779 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:36.779 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:36.779 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:36.779 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:36.846 [initandlisten] connection accepted from 127.0.0.5:44719 #77 (8 connections now open) m31001| Fri Jan 18 13:11:36.846 [conn76] end connection 127.0.0.5:45911 (7 connections now open) m31003| Fri Jan 18 13:11:36.970 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:36.970 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:36.970 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:36.970 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:37.052 [conn77] end connection 127.0.0.5:44719 (6 connections now open) m31001| Fri Jan 18 13:11:37.060 [initandlisten] connection accepted from 127.0.0.5:58299 #78 (7 connections now open) m31003| Fri Jan 18 13:11:37.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:37.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:37.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:37.176 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:37.187 [conn78] end connection 127.0.0.5:58299 (6 connections now open) m31001| Fri Jan 18 13:11:37.275 [initandlisten] connection accepted from 127.0.0.5:51570 #79 (7 connections now open) m31003| Fri Jan 18 13:11:37.358 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:37.358 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:37.358 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:37.358 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31004| Fri Jan 18 13:11:37.392 [conn23] end connection 127.0.0.3:42430 (4 connections now open) m31004| Fri Jan 18 13:11:37.392 [initandlisten] connection accepted from 127.0.0.3:36333 #27 (5 connections now open) m31001| Fri Jan 18 13:11:37.413 [conn79] end connection 127.0.0.5:51570 (6 connections now open) m31003| Fri Jan 18 13:11:37.441 [conn22] end connection 127.0.0.6:45510 (3 connections now open) m31003| Fri Jan 18 13:11:37.441 [initandlisten] connection accepted from 127.0.0.6:47817 #25 (4 connections now open) m31001| Fri Jan 18 13:11:37.477 [initandlisten] connection accepted from 127.0.0.5:36137 #80 (7 connections now open) m31003| Fri Jan 18 13:11:37.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:37.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:37.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:37.524 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:37.558 [conn80] end connection 127.0.0.5:36137 (6 connections now open) m31001| Fri Jan 18 13:11:37.639 [initandlisten] connection accepted from 127.0.0.5:53625 #81 (7 connections now open) m31003| Fri Jan 18 13:11:37.753 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:37.753 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:37.753 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:37.753 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:37.793 [conn81] end connection 127.0.0.5:53625 (6 connections now open) m31001| Fri Jan 18 13:11:37.865 [initandlisten] connection accepted from 127.0.0.5:36874 #82 (7 connections now open) m31001| Fri Jan 18 13:11:37.983 [conn82] end connection 127.0.0.5:36874 (6 connections now open) m31003| Fri Jan 18 13:11:37.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:37.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:37.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:37.985 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:37.996 [initandlisten] connection accepted from 127.0.0.5:53159 #83 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 188, "optime" : { "t" : 1358532698000, "i" : 238 }, "optimeDate" : ISODate("2013-01-18T18:11:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532696000, "i" : 3560 }, "optimeDate" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 153, "optime" : { "t" : 1358532697000, "i" : 4199 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 188, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532588000, "i" : 3441 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532697000, "i" : 4191 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 152, "optime" : { "t" : 1358532696000, "i" : 933 }, "optimeDate" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 152, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 187, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 152, "optime" : { "t" : 1358532588000, "i" : 2741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:36Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 152, "optime" : { "t" : 1358532696000, "i" : 909 }, "optimeDate" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:38Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 21 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 34 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 186, "optime" : { "t" : 1358532588000, "i" : 3741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 154, "optime" : { "t" : 1358532695000, "i" : 542 }, "optimeDate" : ISODate("2013-01-18T18:11:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:37Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.072 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.072 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:38.072 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:38.072 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 153, "optime" : { "t" : 1358532697000, "i" : 4547 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532626000, "i" : 10003 }, "optimeDate" : ISODate("2013-01-18T18:10:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 153, "optime" : { "t" : 1358532588000, "i" : 3441 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 186, "optime" : { "t" : 1358532698000, "i" : 238 }, "optimeDate" : ISODate("2013-01-18T18:11:38Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31004| Fri Jan 18 13:11:38.090 [conn24] end connection 127.0.0.4:40990 (4 connections now open) m31004| Fri Jan 18 13:11:38.090 [initandlisten] connection accepted from 127.0.0.4:49903 #28 (6 connections now open) m31001| Fri Jan 18 13:11:38.140 [conn83] end connection 127.0.0.5:53159 (6 connections now open) m31001| Fri Jan 18 13:11:38.140 [initandlisten] connection accepted from 127.0.0.5:46101 #84 (8 connections now open) m31003| Fri Jan 18 13:11:38.270 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.270 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:38.270 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:38.270 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:38.298 [conn84] end connection 127.0.0.5:46101 (6 connections now open) m31001| Fri Jan 18 13:11:38.318 [initandlisten] connection accepted from 127.0.0.5:55214 #85 (7 connections now open) m31000| Fri Jan 18 13:11:38.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Fri Jan 18 13:11:38.430 [conn85] end connection 127.0.0.5:55214 (6 connections now open) m31003| Fri Jan 18 13:11:38.434 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.434 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:38.434 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:38.435 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:11:38.437 [conn26] end connection 127.0.0.2:40874 (4 connections now open) m31002| Fri Jan 18 13:11:38.437 [initandlisten] connection accepted from 127.0.0.2:56843 #30 (5 connections now open) m31001| Fri Jan 18 13:11:38.563 [initandlisten] connection accepted from 127.0.0.5:34466 #86 (7 connections now open) m31003| Fri Jan 18 13:11:38.711 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.711 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:38.711 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:38.711 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:38.729 [conn86] end connection 127.0.0.5:34466 (6 connections now open) m31001| Fri Jan 18 13:11:38.769 [initandlisten] connection accepted from 127.0.0.5:43412 #87 (7 connections now open) m31003| Fri Jan 18 13:11:38.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:38.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:38.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:38.839 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:38.919 [initandlisten] connection accepted from 127.0.0.5:50762 #88 (8 connections now open) m31001| Fri Jan 18 13:11:38.919 [conn87] end connection 127.0.0.5:43412 (7 connections now open) m31003| Fri Jan 18 13:11:39.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:39.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:39.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:39.058 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:39.141 [conn88] end connection 127.0.0.5:50762 (6 connections now open) m31001| Fri Jan 18 13:11:39.213 [initandlisten] connection accepted from 127.0.0.5:37827 #89 (7 connections now open) m31003| Fri Jan 18 13:11:39.356 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:39.356 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:39.356 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:39.356 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:11:39.393 [conn27] end connection 127.0.0.3:46440 (4 connections now open) m31002| Fri Jan 18 13:11:39.393 [initandlisten] connection accepted from 127.0.0.3:42792 #31 (5 connections now open) m31001| Fri Jan 18 13:11:39.441 [conn26] end connection 127.0.0.6:60540 (6 connections now open) m31001| Fri Jan 18 13:11:39.441 [initandlisten] connection accepted from 127.0.0.6:38465 #90 (7 connections now open) m31001| Fri Jan 18 13:11:39.453 [initandlisten] connection accepted from 127.0.0.5:48688 #91 (8 connections now open) m31001| Fri Jan 18 13:11:39.453 [conn89] end connection 127.0.0.5:37827 (7 connections now open) m31003| Fri Jan 18 13:11:39.574 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:39.574 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:39.574 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:39.574 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:39.585 [conn91] end connection 127.0.0.5:48688 (6 connections now open) m31001| Fri Jan 18 13:11:39.599 [initandlisten] connection accepted from 127.0.0.5:55760 #92 (7 connections now open) m31001| Fri Jan 18 13:11:39.692 [conn92] end connection 127.0.0.5:55760 (6 connections now open) m31003| Fri Jan 18 13:11:39.701 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:39.701 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:39.701 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:39.701 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:39.721 [initandlisten] connection accepted from 127.0.0.5:49936 #93 (7 connections now open) m31001| Fri Jan 18 13:11:39.773 [conn93] end connection 127.0.0.5:49936 (6 connections now open) m31003| Fri Jan 18 13:11:39.776 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:39.776 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:39.776 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:39.776 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:39.961 [initandlisten] connection accepted from 127.0.0.5:48883 #94 (7 connections now open) m31001| Fri Jan 18 13:11:40.062 [conn94] end connection 127.0.0.5:48883 (6 connections now open) m31003| Fri Jan 18 13:11:40.065 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:40.065 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.065 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:40.065 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:40.090 [conn27] end connection 127.0.0.4:52753 (5 connections now open) m31001| Fri Jan 18 13:11:40.090 [initandlisten] connection accepted from 127.0.0.4:42201 #95 (6 connections now open) m31001| Fri Jan 18 13:11:40.111 [initandlisten] connection accepted from 127.0.0.5:52484 #96 (7 connections now open) m31003| Fri Jan 18 13:11:40.220 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:40.220 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.220 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:40.220 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:40.312 [initandlisten] connection accepted from 127.0.0.5:60256 #97 (8 connections now open) m31001| Fri Jan 18 13:11:40.312 [conn96] end connection 127.0.0.5:52484 (7 connections now open) m31003| Fri Jan 18 13:11:40.487 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:40.487 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.487 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:40.487 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:40.521 [conn97] end connection 127.0.0.5:60256 (6 connections now open) m31001| Fri Jan 18 13:11:40.570 [conn28] end connection 127.0.0.5:36771 (5 connections now open) m31001| Fri Jan 18 13:11:40.635 [initandlisten] connection accepted from 127.0.0.5:44984 #98 (6 connections now open) m31001| Fri Jan 18 13:11:40.635 [initandlisten] connection accepted from 127.0.0.5:37998 #99 (7 connections now open) m31003| Fri Jan 18 13:11:40.710 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:40.710 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.710 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.710 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:11:40.752 [initandlisten] connection accepted from 127.0.0.5:40084 #32 (6 connections now open) m31001| Fri Jan 18 13:11:40.802 [conn99] end connection 127.0.0.5:37998 (6 connections now open) m31002| Fri Jan 18 13:11:40.960 [conn32] end connection 127.0.0.5:40084 (5 connections now open) m31003| Fri Jan 18 13:11:40.963 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:40.963 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.963 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:40.963 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.022 [initandlisten] connection accepted from 127.0.0.5:44366 #33 (6 connections now open) m31001| Fri Jan 18 13:11:41.027 [conn35] end connection 127.0.0.5:38271 (5 connections now open) m31002| Fri Jan 18 13:11:41.062 [initandlisten] connection accepted from 127.0.0.5:56891 #34 (7 connections now open) m31003| Fri Jan 18 13:11:41.132 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.132 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.132 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.158 [initandlisten] connection accepted from 127.0.0.5:40914 #35 (8 connections now open) m31002| Fri Jan 18 13:11:41.158 [conn33] end connection 127.0.0.5:44366 (7 connections now open) m31002| Fri Jan 18 13:11:41.192 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Fri Jan 18 13:11:41.192 [initandlisten] connection accepted from 127.0.0.4:58455 #47 (12 connections now open) m31003| Fri Jan 18 13:11:41.315 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.315 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.315 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.333 [conn34] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532588000|5241 } } cursorid:804517902846325 ntoreturn:0 keyUpdates:0 numYields: 2 locks(micros) r:255984 nreturned:162624 reslen:2764628 141ms m31002| Fri Jan 18 13:11:41.337 [initandlisten] connection accepted from 127.0.0.5:55693 #36 (8 connections now open) m31002| Fri Jan 18 13:11:41.337 [conn35] end connection 127.0.0.5:40914 (7 connections now open) m31003| Fri Jan 18 13:11:41.497 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.497 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.497 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.516 [initandlisten] connection accepted from 127.0.0.5:36252 #37 (8 connections now open) m31002| Fri Jan 18 13:11:41.516 [conn36] end connection 127.0.0.5:55693 (7 connections now open) m31003| Fri Jan 18 13:11:41.636 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.636 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.636 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.644 [conn37] end connection 127.0.0.5:36252 (6 connections now open) m31002| Fri Jan 18 13:11:41.712 [initandlisten] connection accepted from 127.0.0.5:60211 #38 (7 connections now open) m31003| Fri Jan 18 13:11:41.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.839 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.853 [conn38] end connection 127.0.0.5:60211 (6 connections now open) m31003| Fri Jan 18 13:11:41.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Fri Jan 18 13:11:41.871 [initandlisten] connection accepted from 127.0.0.5:59847 #39 (7 connections now open) m31003| Fri Jan 18 13:11:41.971 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:41.971 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:41.971 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:41.973 [initandlisten] connection accepted from 127.0.0.5:36353 #40 (8 connections now open) m31002| Fri Jan 18 13:11:41.973 [conn39] end connection 127.0.0.5:59847 (7 connections now open) m31003| Fri Jan 18 13:11:42.148 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:42.148 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:42.148 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:42.193 [slaveTracking] build index local.slaves { _id: 1 } m31002| Fri Jan 18 13:11:42.193 [slaveTracking] build index done. scanned 0 total records. 0 secs m31002| Fri Jan 18 13:11:42.201 [conn40] end connection 127.0.0.5:36353 (6 connections now open) m31002| Fri Jan 18 13:11:42.255 [initandlisten] connection accepted from 127.0.0.5:51408 #41 (7 connections now open) m31002| Fri Jan 18 13:11:42.368 [conn41] end connection 127.0.0.5:51408 (6 connections now open) m31003| Fri Jan 18 13:11:42.370 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:42.370 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:42.370 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:42.437 [initandlisten] connection accepted from 127.0.0.5:40691 #42 (7 connections now open) m31003| Fri Jan 18 13:11:42.554 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:42.554 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:42.554 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:42.621 [conn42] end connection 127.0.0.5:40691 (6 connections now open) m31002| Fri Jan 18 13:11:42.664 [initandlisten] connection accepted from 127.0.0.5:60440 #43 (7 connections now open) m31003| Fri Jan 18 13:11:42.719 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:42.719 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:42.719 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:42.719 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:11:42.719 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Fri Jan 18 13:11:42.780 [conn43] end connection 127.0.0.5:60440 (6 connections now open) m31001| Fri Jan 18 13:11:42.807 [initandlisten] connection accepted from 127.0.0.5:50772 #100 (6 connections now open) m31003| Fri Jan 18 13:11:42.910 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:42.910 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:42.910 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:42.910 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:11:42.910 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31001| Fri Jan 18 13:11:42.930 [conn100] end connection 127.0.0.5:50772 (5 connections now open) m31001| Fri Jan 18 13:11:42.980 [initandlisten] connection accepted from 127.0.0.5:35292 #101 (6 connections now open) m31001| Fri Jan 18 13:11:42.980 [initandlisten] connection accepted from 127.0.0.5:34448 #102 (7 connections now open) m31002| Fri Jan 18 13:11:43.004 [conn34] end connection 127.0.0.5:56891 (5 connections now open) m31003| Fri Jan 18 13:11:43.088 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.089 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.089 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.089 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.114 [conn101] end connection 127.0.0.5:35292 (6 connections now open) m31001| Fri Jan 18 13:11:43.177 [initandlisten] connection accepted from 127.0.0.5:59280 #103 (7 connections now open) m31000| Fri Jan 18 13:11:43.224 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:102227 102ms m31003| Fri Jan 18 13:11:43.240 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.240 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.240 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.240 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.249 [conn102] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532588000|6341 } } cursorid:815563525274711 ntoreturn:0 keyUpdates:0 numYields: 2 locks(micros) r:244907 nreturned:167241 reslen:2843117 134ms STATUS FROM : 127.0.0.2:31000 m31001| Fri Jan 18 13:11:43.280 [initandlisten] connection accepted from 127.0.0.5:60462 #104 (8 connections now open) m31001| Fri Jan 18 13:11:43.280 [conn103] end connection 127.0.0.5:59280 (7 connections now open) m31000| Fri Jan 18 13:11:43.330 [conn46] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532587000|501 } } cursorid:756989324115364 ntoreturn:0 keyUpdates:0 numYields: 1844 locks(micros) r:408381 nreturned:237216 reslen:4032692 208ms m31003| Fri Jan 18 13:11:43.333 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.333 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.333 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.333 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.362 [initandlisten] connection accepted from 127.0.0.5:41980 #105 (8 connections now open) m31003| Fri Jan 18 13:11:43.414 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.414 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.414 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.414 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.426 [initandlisten] connection accepted from 127.0.0.5:56504 #106 (9 connections now open) m31001| Fri Jan 18 13:11:43.426 [conn104] end connection 127.0.0.5:60462 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 193, "optime" : { "t" : 1358532703000, "i" : 2847 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532702000, "i" : 3510 }, "optimeDate" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 158, "optime" : { "t" : 1358532703000, "i" : 1795 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 193, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532588000, "i" : 6641 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532703000, "i" : 1692 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 157, "optime" : { "t" : 1358532702000, "i" : 445 }, "optimeDate" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 192, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358532588000, "i" : 5841 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358532702000, "i" : 442 }, "optimeDate" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 m31001| Fri Jan 18 13:11:43.453 [conn105] end connection 127.0.0.5:41980 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:43Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 24 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 37 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532588000, "i" : 6741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 159, "optime" : { "t" : 1358532703000, "i" : 1692 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:43Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.524 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 158, "optime" : { "t" : 1358532703000, "i" : 2631 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1358532588000, "i" : 6741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532703000, "i" : 2847 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:11:43.591 [conn106] end connection 127.0.0.5:56504 (6 connections now open) m31001| Fri Jan 18 13:11:43.612 [initandlisten] connection accepted from 127.0.0.5:50248 #107 (7 connections now open) m31003| Fri Jan 18 13:11:43.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.714 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.714 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.715 [conn107] end connection 127.0.0.5:50248 (6 connections now open) m31001| Fri Jan 18 13:11:43.715 [initandlisten] connection accepted from 127.0.0.5:44746 #108 (8 connections now open) m31003| Fri Jan 18 13:11:43.862 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:43.862 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:43.862 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:43.862 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:43.897 [conn108] end connection 127.0.0.5:44746 (6 connections now open) m31001| Fri Jan 18 13:11:43.897 [initandlisten] connection accepted from 127.0.0.5:34734 #109 (8 connections now open) m31001| Fri Jan 18 13:11:44.040 [conn109] end connection 127.0.0.5:34734 (6 connections now open) m31003| Fri Jan 18 13:11:44.044 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:44.044 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:44.044 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:44.044 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:44.055 [initandlisten] connection accepted from 127.0.0.5:57307 #110 (7 connections now open) m31001| Fri Jan 18 13:11:44.254 [conn110] end connection 127.0.0.5:57307 (6 connections now open) m31003| Fri Jan 18 13:11:44.257 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:44.257 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:44.257 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:44.257 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:44.349 [initandlisten] connection accepted from 127.0.0.5:48507 #111 (7 connections now open) m31003| Fri Jan 18 13:11:44.476 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:44.476 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:44.476 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:44.476 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:44.511 [conn111] end connection 127.0.0.5:48507 (6 connections now open) m31001| Fri Jan 18 13:11:44.596 [initandlisten] connection accepted from 127.0.0.5:39046 #112 (7 connections now open) m31003| Fri Jan 18 13:11:44.736 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:44.736 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:44.736 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:44.736 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:44.753 [conn112] end connection 127.0.0.5:39046 (6 connections now open) m31001| Fri Jan 18 13:11:44.846 [initandlisten] connection accepted from 127.0.0.5:46403 #113 (7 connections now open) m31001| Fri Jan 18 13:11:44.891 [conn113] end connection 127.0.0.5:46403 (6 connections now open) m31003| Fri Jan 18 13:11:44.894 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:44.894 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:44.894 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:44.894 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:44.912 [initandlisten] connection accepted from 127.0.0.5:38674 #114 (7 connections now open) m31003| Fri Jan 18 13:11:45.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.024 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.027 [conn114] end connection 127.0.0.5:38674 (6 connections now open) m31001| Fri Jan 18 13:11:45.027 [initandlisten] connection accepted from 127.0.0.5:43614 #115 (8 connections now open) m31003| Fri Jan 18 13:11:45.160 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.160 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.160 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.160 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.187 [conn115] end connection 127.0.0.5:43614 (6 connections now open) m31001| Fri Jan 18 13:11:45.187 [initandlisten] connection accepted from 127.0.0.5:46096 #116 (8 connections now open) m31003| Fri Jan 18 13:11:45.337 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.337 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.337 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.337 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.405 [conn116] end connection 127.0.0.5:46096 (6 connections now open) m31001| Fri Jan 18 13:11:45.405 [initandlisten] connection accepted from 127.0.0.5:57910 #117 (8 connections now open) m31001| Fri Jan 18 13:11:45.540 [conn117] end connection 127.0.0.5:57910 (6 connections now open) m31003| Fri Jan 18 13:11:45.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.543 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.595 [initandlisten] connection accepted from 127.0.0.5:32969 #118 (7 connections now open) m31003| Fri Jan 18 13:11:45.744 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.744 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.744 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.744 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.761 [conn118] end connection 127.0.0.5:32969 (6 connections now open) m31001| Fri Jan 18 13:11:45.828 [initandlisten] connection accepted from 127.0.0.5:38866 #119 (7 connections now open) m31003| Fri Jan 18 13:11:45.902 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:45.902 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:45.902 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:45.902 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:45.913 [conn119] end connection 127.0.0.5:38866 (6 connections now open) m31001| Fri Jan 18 13:11:45.987 [initandlisten] connection accepted from 127.0.0.5:39308 #120 (7 connections now open) m31003| Fri Jan 18 13:11:46.191 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:46.191 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:46.191 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:46.191 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:46.268 [conn120] end connection 127.0.0.5:39308 (6 connections now open) m31001| Fri Jan 18 13:11:46.320 [initandlisten] connection accepted from 127.0.0.5:42859 #121 (7 connections now open) m31001| Fri Jan 18 13:11:46.412 [conn121] end connection 127.0.0.5:42859 (6 connections now open) m31003| Fri Jan 18 13:11:46.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:46.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:46.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:46.416 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:46.569 [initandlisten] connection accepted from 127.0.0.5:42216 #122 (7 connections now open) m31003| Fri Jan 18 13:11:46.630 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:46.631 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:46.631 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:46.631 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:46.708 [conn122] end connection 127.0.0.5:42216 (6 connections now open) m31001| Fri Jan 18 13:11:46.760 [initandlisten] connection accepted from 127.0.0.5:50222 #123 (7 connections now open) m31003| Fri Jan 18 13:11:46.952 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:46.952 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:46.952 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:11:46.952 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:11:46.952 [initandlisten] connection accepted from 127.0.0.5:55563 #44 (6 connections now open) m31001| Fri Jan 18 13:11:46.987 [conn123] end connection 127.0.0.5:50222 (6 connections now open) m31003| Fri Jan 18 13:11:47.106 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Fri Jan 18 13:11:47.106 [conn102] end connection 127.0.0.5:34448 (5 connections now open) m31002| Fri Jan 18 13:11:47.158 [initandlisten] connection accepted from 127.0.0.5:54976 #45 (7 connections now open) m31002| Fri Jan 18 13:11:47.158 [conn44] end connection 127.0.0.5:55563 (5 connections now open) m31003| Fri Jan 18 13:11:47.158 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:47.158 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:47.158 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:47.250 [initandlisten] connection accepted from 127.0.0.5:47735 #46 (7 connections now open) m31003| Fri Jan 18 13:11:47.306 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:47.306 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:47.306 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:47.386 [conn46] end connection 127.0.0.5:47735 (6 connections now open) m31002| Fri Jan 18 13:11:47.481 [initandlisten] connection accepted from 127.0.0.5:41786 #47 (7 connections now open) m31000| Fri Jan 18 13:11:47.513 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:102400 102ms m31002| Fri Jan 18 13:11:47.518 [conn45] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532588000|8641 } } cursorid:830535157228573 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:172985 nreturned:164941 reslen:2804017 132ms m31003| Fri Jan 18 13:11:47.529 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:47.529 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:47.529 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31000| Fri Jan 18 13:11:47.598 [conn47] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532588000|5341 } } cursorid:810735812345093 ntoreturn:0 keyUpdates:0 numYields: 1884 locks(micros) r:411259 nreturned:241032 reslen:4097564 212ms m31002| Fri Jan 18 13:11:47.614 [conn47] end connection 127.0.0.5:41786 (6 connections now open) m31002| Fri Jan 18 13:11:47.666 [initandlisten] connection accepted from 127.0.0.5:40085 #48 (7 connections now open) m31003| Fri Jan 18 13:11:47.730 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:47.730 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:47.730 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:47.755 [conn48] end connection 127.0.0.5:40085 (6 connections now open) m31002| Fri Jan 18 13:11:47.837 [initandlisten] connection accepted from 127.0.0.5:43013 #49 (7 connections now open) m31003| Fri Jan 18 13:11:47.942 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:47.942 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:47.942 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:47.973 [initandlisten] connection accepted from 127.0.0.5:42001 #50 (8 connections now open) m31002| Fri Jan 18 13:11:48.003 [conn49] end connection 127.0.0.5:43013 (7 connections now open) m31003| Fri Jan 18 13:11:48.130 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.130 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.130 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:48.214 [initandlisten] connection accepted from 127.0.0.5:57805 #51 (8 connections now open) m31002| Fri Jan 18 13:11:48.214 [conn50] end connection 127.0.0.5:42001 (6 connections now open) m31003| Fri Jan 18 13:11:48.266 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.266 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.266 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:48.280 [conn51] end connection 127.0.0.5:57805 (6 connections now open) m31002| Fri Jan 18 13:11:48.335 [initandlisten] connection accepted from 127.0.0.5:53703 #52 (7 connections now open) m31003| Fri Jan 18 13:11:48.384 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.384 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.384 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:48.422 [conn52] end connection 127.0.0.5:53703 (6 connections now open) m31002| Fri Jan 18 13:11:48.453 [initandlisten] connection accepted from 127.0.0.5:48316 #53 (7 connections now open) m31002| Fri Jan 18 13:11:48.506 [conn53] end connection 127.0.0.5:48316 (6 connections now open) m31003| Fri Jan 18 13:11:48.508 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.508 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.508 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:48.587 [initandlisten] connection accepted from 127.0.0.5:59598 #54 (7 connections now open) m31002| Fri Jan 18 13:11:48.651 [conn54] end connection 127.0.0.5:59598 (6 connections now open) m31003| Fri Jan 18 13:11:48.660 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.660 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.660 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:48.660 [initandlisten] connection accepted from 127.0.0.5:58093 #55 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 198, "optime" : { "t" : 1358532708000, "i" : 5653 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532708000, "i" : 3427 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 163, "optime" : { "t" : 1358532707000, "i" : 3256 }, "optimeDate" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 198, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532588000, "i" : 8741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532707000, "i" : 3256 }, "optimeDate" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 162, "optime" : { "t" : 1358532708000, "i" : 613 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 197, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358532589000, "i" : 164 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358532708000, "i" : 613 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:48Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 22 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 23 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532589000, "i" : 664 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 164, "optime" : { "t" : 1358532708000, "i" : 3066 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.730 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.730 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.730 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:48.730 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:11:48.730 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 163, "optime" : { "t" : 1358532707000, "i" : 3256 }, "optimeDate" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 163, "optime" : { "t" : 1358532588000, "i" : 8741 }, "optimeDate" : ISODate("2013-01-18T18:09:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532708000, "i" : 5653 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:11:48.774 [conn55] end connection 127.0.0.5:58093 (6 connections now open) m31001| Fri Jan 18 13:11:48.808 [initandlisten] connection accepted from 127.0.0.5:51052 #124 (6 connections now open) m31003| Fri Jan 18 13:11:48.942 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:48.942 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:11:48.942 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:48.942 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:48.942 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:11:48.943 [conn45] end connection 127.0.0.5:54976 (5 connections now open) m31001| Fri Jan 18 13:11:48.985 [initandlisten] connection accepted from 127.0.0.5:33006 #125 (7 connections now open) m31001| Fri Jan 18 13:11:48.985 [conn124] end connection 127.0.0.5:51052 (5 connections now open) m31001| Fri Jan 18 13:11:49.003 [initandlisten] connection accepted from 127.0.0.5:35078 #126 (7 connections now open) m31003| Fri Jan 18 13:11:49.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:49.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:49.176 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:49.176 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:49.257 [conn126] end connection 127.0.0.5:35078 (6 connections now open) m31001| Fri Jan 18 13:11:49.271 [initandlisten] connection accepted from 127.0.0.5:35033 #127 (7 connections now open) m31001| Fri Jan 18 13:11:49.389 [conn125] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532589000|864 } } cursorid:841333165304964 ntoreturn:0 keyUpdates:0 numYields: 3 locks(micros) r:265262 nreturned:163841 reslen:2785317 134ms m31003| Fri Jan 18 13:11:49.398 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:49.398 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:49.398 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:49.398 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:49.430 [conn127] end connection 127.0.0.5:35033 (6 connections now open) m31001| Fri Jan 18 13:11:49.430 [initandlisten] connection accepted from 127.0.0.5:50918 #128 (8 connections now open) m31003| Fri Jan 18 13:11:49.523 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:49.523 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:49.523 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:49.523 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:49.536 [initandlisten] connection accepted from 127.0.0.5:60224 #129 (8 connections now open) m31001| Fri Jan 18 13:11:49.574 [conn128] end connection 127.0.0.5:50918 (7 connections now open) m31003| Fri Jan 18 13:11:49.680 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:49.680 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:49.680 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:49.680 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:49.762 [conn129] end connection 127.0.0.5:60224 (6 connections now open) m31001| Fri Jan 18 13:11:49.762 [initandlisten] connection accepted from 127.0.0.5:37815 #130 (8 connections now open) m31001| Fri Jan 18 13:11:49.809 [conn130] end connection 127.0.0.5:37815 (6 connections now open) m31003| Fri Jan 18 13:11:49.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:49.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:49.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:49.811 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:49.952 [initandlisten] connection accepted from 127.0.0.5:56700 #131 (7 connections now open) m31001| Fri Jan 18 13:11:50.063 [conn131] end connection 127.0.0.5:56700 (6 connections now open) m31003| Fri Jan 18 13:11:50.066 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:50.066 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.066 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:50.066 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:50.072 [initandlisten] connection accepted from 127.0.0.5:52701 #132 (7 connections now open) m31001| Fri Jan 18 13:11:50.206 [conn132] end connection 127.0.0.5:52701 (6 connections now open) m31003| Fri Jan 18 13:11:50.210 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:50.210 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.211 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:50.211 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:50.263 [initandlisten] connection accepted from 127.0.0.5:55512 #133 (7 connections now open) m31003| Fri Jan 18 13:11:50.367 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:50.367 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.367 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:50.367 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:50.410 [conn133] end connection 127.0.0.5:55512 (6 connections now open) m31001| Fri Jan 18 13:11:50.426 [initandlisten] connection accepted from 127.0.0.5:37392 #134 (7 connections now open) m31003| Fri Jan 18 13:11:50.595 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:50.595 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.595 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:50.595 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:50.688 [initandlisten] connection accepted from 127.0.0.5:35324 #135 (8 connections now open) m31001| Fri Jan 18 13:11:50.688 [conn134] end connection 127.0.0.5:37392 (7 connections now open) m31003| Fri Jan 18 13:11:50.835 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:50.835 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.835 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:11:50.835 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:11:50.835 [initandlisten] connection accepted from 127.0.0.5:40742 #56 (6 connections now open) m31001| Fri Jan 18 13:11:50.868 [conn135] end connection 127.0.0.5:35324 (6 connections now open) m31003| Fri Jan 18 13:11:51.015 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.015 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.015 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.015 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31001| Fri Jan 18 13:11:51.015 [conn125] end connection 127.0.0.5:33006 (5 connections now open) m31002| Fri Jan 18 13:11:51.018 [conn56] end connection 127.0.0.5:40742 (5 connections now open) m31002| Fri Jan 18 13:11:51.046 [initandlisten] connection accepted from 127.0.0.5:38685 #57 (6 connections now open) m31002| Fri Jan 18 13:11:51.108 [initandlisten] connection accepted from 127.0.0.5:47157 #58 (7 connections now open) m31003| Fri Jan 18 13:11:51.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.241 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:51.284 [conn58] end connection 127.0.0.5:47157 (6 connections now open) m31002| Fri Jan 18 13:11:51.299 [initandlisten] connection accepted from 127.0.0.5:49916 #59 (7 connections now open) m31002| Fri Jan 18 13:11:51.434 [conn57] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532589000|1964 } } cursorid:846986407180197 ntoreturn:0 keyUpdates:0 numYields: 2 locks(micros) r:256900 nreturned:162741 reslen:2766617 134ms m31003| Fri Jan 18 13:11:51.446 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.446 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.446 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:51.493 [initandlisten] connection accepted from 127.0.0.5:36491 #60 (8 connections now open) m31002| Fri Jan 18 13:11:51.515 [conn59] end connection 127.0.0.5:49916 (7 connections now open) m31003| Fri Jan 18 13:11:51.566 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.566 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.566 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:51.631 [conn60] end connection 127.0.0.5:36491 (6 connections now open) m31002| Fri Jan 18 13:11:51.655 [initandlisten] connection accepted from 127.0.0.5:38007 #61 (7 connections now open) m31003| Fri Jan 18 13:11:51.786 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.786 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.786 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:51.850 [conn61] end connection 127.0.0.5:38007 (6 connections now open) m31003| Fri Jan 18 13:11:51.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Fri Jan 18 13:11:51.891 [initandlisten] connection accepted from 127.0.0.5:51010 #62 (7 connections now open) m31003| Fri Jan 18 13:11:51.938 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:51.938 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:51.938 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:52.012 [initandlisten] connection accepted from 127.0.0.5:40503 #63 (8 connections now open) m31002| Fri Jan 18 13:11:52.012 [conn62] end connection 127.0.0.5:51010 (6 connections now open) m31003| Fri Jan 18 13:11:52.215 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:52.215 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:52.215 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:52.258 [conn63] end connection 127.0.0.5:40503 (6 connections now open) m31002| Fri Jan 18 13:11:52.325 [initandlisten] connection accepted from 127.0.0.5:57770 #64 (7 connections now open) m31003| Fri Jan 18 13:11:52.412 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:52.412 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:52.412 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:52.439 [conn64] end connection 127.0.0.5:57770 (6 connections now open) m31002| Fri Jan 18 13:11:52.439 [initandlisten] connection accepted from 127.0.0.5:56564 #65 (8 connections now open) m31002| Fri Jan 18 13:11:52.589 [conn65] end connection 127.0.0.5:56564 (6 connections now open) m31003| Fri Jan 18 13:11:52.597 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:52.597 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:52.597 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:52.717 [initandlisten] connection accepted from 127.0.0.5:49206 #66 (7 connections now open) m31003| Fri Jan 18 13:11:52.830 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:52.830 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:52.830 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:52.879 [conn66] end connection 127.0.0.5:49206 (6 connections now open) m31002| Fri Jan 18 13:11:52.970 [initandlisten] connection accepted from 127.0.0.5:48903 #67 (7 connections now open) m31003| Fri Jan 18 13:11:53.055 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:53.055 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:53.055 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:53.114 [conn67] end connection 127.0.0.5:48903 (6 connections now open) m31002| Fri Jan 18 13:11:53.214 [initandlisten] connection accepted from 127.0.0.5:50721 #68 (7 connections now open) m31003| Fri Jan 18 13:11:53.330 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:53.330 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:53.330 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:53.357 [initandlisten] connection accepted from 127.0.0.5:60463 #69 (8 connections now open) m31002| Fri Jan 18 13:11:53.357 [conn68] end connection 127.0.0.5:50721 (6 connections now open) m31003| Fri Jan 18 13:11:53.394 [conn23] end connection 127.0.0.3:40368 (3 connections now open) m31003| Fri Jan 18 13:11:53.394 [initandlisten] connection accepted from 127.0.0.3:43485 #26 (4 connections now open) m31002| Fri Jan 18 13:11:53.443 [conn28] end connection 127.0.0.6:38590 (6 connections now open) m31002| Fri Jan 18 13:11:53.443 [initandlisten] connection accepted from 127.0.0.6:53787 #70 (8 connections now open) m31003| Fri Jan 18 13:11:53.495 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:53.495 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:53.495 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:53.584 [conn69] end connection 127.0.0.5:60463 (6 connections now open) m31002| Fri Jan 18 13:11:53.607 [initandlisten] connection accepted from 127.0.0.5:50941 #71 (7 connections now open) m31003| Fri Jan 18 13:11:53.656 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:53.656 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:53.656 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:53.657 [initandlisten] connection accepted from 127.0.0.5:48902 #72 (8 connections now open) m31002| Fri Jan 18 13:11:53.657 [conn71] end connection 127.0.0.5:50941 (7 connections now open) m31003| Fri Jan 18 13:11:53.785 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:53.785 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:53.785 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:53.834 [conn72] end connection 127.0.0.5:48902 (6 connections now open) m31002| Fri Jan 18 13:11:53.865 [initandlisten] connection accepted from 127.0.0.5:54873 #73 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 203, "optime" : { "t" : 1358532713000, "i" : 9004 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532712000, "i" : 3585 }, "optimeDate" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 168, "optime" : { "t" : 1358532713000, "i" : 2796 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 203, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 168, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 168, "optime" : { "t" : 1358532589000, "i" : 3064 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 168, "optime" : { "t" : 1358532713000, "i" : 2796 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 167, "optime" : { "t" : 1358532712000, "i" : 586 }, "optimeDate" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 202, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358532589000, "i" : 2464 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358532712000, "i" : 453 }, "optimeDate" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:53Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 24 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 13 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532589000, "i" : 3364 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532713000, "i" : 2796 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:53Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.004 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.004 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:54.004 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 169, "optime" : { "t" : 1358532713000, "i" : 2796 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 169, "optime" : { "t" : 1358532589000, "i" : 3064 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 202, "optime" : { "t" : 1358532713000, "i" : 9004 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:11:54.065 [conn73] end connection 127.0.0.5:54873 (6 connections now open) m31002| Fri Jan 18 13:11:54.070 [initandlisten] connection accepted from 127.0.0.5:44671 #74 (7 connections now open) m31003| Fri Jan 18 13:11:54.092 [conn24] end connection 127.0.0.4:52017 (3 connections now open) m31003| Fri Jan 18 13:11:54.092 [initandlisten] connection accepted from 127.0.0.4:41882 #27 (4 connections now open) m31003| Fri Jan 18 13:11:54.214 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.214 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:54.214 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:54.219 [conn74] end connection 127.0.0.5:44671 (6 connections now open) m31002| Fri Jan 18 13:11:54.334 [initandlisten] connection accepted from 127.0.0.5:40982 #75 (7 connections now open) m31001| Fri Jan 18 13:11:54.440 [conn29] end connection 127.0.0.2:36052 (4 connections now open) m31001| Fri Jan 18 13:11:54.440 [initandlisten] connection accepted from 127.0.0.2:60375 #136 (5 connections now open) m31003| Fri Jan 18 13:11:54.447 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.447 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:54.447 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:54.525 [conn75] end connection 127.0.0.5:40982 (6 connections now open) m31002| Fri Jan 18 13:11:54.613 [initandlisten] connection accepted from 127.0.0.5:58371 #76 (7 connections now open) m31002| Fri Jan 18 13:11:54.719 [conn76] end connection 127.0.0.5:58371 (6 connections now open) m31003| Fri Jan 18 13:11:54.722 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.722 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:54.722 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:54.804 [conn29] end connection 127.0.0.5:37355 (5 connections now open) m31002| Fri Jan 18 13:11:54.830 [initandlisten] connection accepted from 127.0.0.5:52244 #77 (6 connections now open) m31002| Fri Jan 18 13:11:54.830 [initandlisten] connection accepted from 127.0.0.5:42860 #78 (7 connections now open) m31003| Fri Jan 18 13:11:54.939 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:54.939 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:54.939 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:54.944 [conn77] end connection 127.0.0.5:52244 (6 connections now open) m31002| Fri Jan 18 13:11:54.953 [initandlisten] connection accepted from 127.0.0.5:47040 #79 (7 connections now open) m31003| Fri Jan 18 13:11:55.151 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:55.151 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:55.151 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:55.157 [conn79] end connection 127.0.0.5:47040 (6 connections now open) m31002| Fri Jan 18 13:11:55.168 [initandlisten] connection accepted from 127.0.0.5:45158 #80 (7 connections now open) m31002| Fri Jan 18 13:11:55.257 [conn80] end connection 127.0.0.5:45158 (6 connections now open) m31003| Fri Jan 18 13:11:55.261 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:55.261 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:55.261 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:55.272 [initandlisten] connection accepted from 127.0.0.5:47467 #81 (7 connections now open) m31000| Fri Jan 18 13:11:55.395 [conn43] end connection 127.0.0.3:41671 (11 connections now open) m31000| Fri Jan 18 13:11:55.395 [initandlisten] connection accepted from 127.0.0.3:36408 #48 (12 connections now open) m31000| Fri Jan 18 13:11:55.443 [conn44] end connection 127.0.0.6:38776 (11 connections now open) m31000| Fri Jan 18 13:11:55.443 [initandlisten] connection accepted from 127.0.0.6:54782 #49 (12 connections now open) m31002| Fri Jan 18 13:11:55.519 [conn81] end connection 127.0.0.5:47467 (6 connections now open) m31003| Fri Jan 18 13:11:55.523 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:55.523 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:55.523 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:55.586 [initandlisten] connection accepted from 127.0.0.5:56559 #82 (7 connections now open) m31003| Fri Jan 18 13:11:55.728 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:55.728 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:55.728 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:55.791 [conn82] end connection 127.0.0.5:56559 (6 connections now open) m31002| Fri Jan 18 13:11:55.791 [initandlisten] connection accepted from 127.0.0.5:57935 #83 (8 connections now open) m31002| Fri Jan 18 13:11:55.929 [conn83] end connection 127.0.0.5:57935 (6 connections now open) m31003| Fri Jan 18 13:11:55.933 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:55.933 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:55.933 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:55.961 [initandlisten] connection accepted from 127.0.0.5:40613 #84 (7 connections now open) m31003| Fri Jan 18 13:11:56.073 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:56.074 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:56.074 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:56.085 [conn84] end connection 127.0.0.5:40613 (6 connections now open) m31000| Fri Jan 18 13:11:56.093 [conn45] end connection 127.0.0.4:35855 (11 connections now open) m31000| Fri Jan 18 13:11:56.093 [initandlisten] connection accepted from 127.0.0.4:58259 #50 (12 connections now open) m31002| Fri Jan 18 13:11:56.134 [initandlisten] connection accepted from 127.0.0.5:38454 #85 (7 connections now open) m31003| Fri Jan 18 13:11:56.210 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:56.211 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:56.211 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:56.253 [conn85] end connection 127.0.0.5:38454 (6 connections now open) m31002| Fri Jan 18 13:11:56.254 [initandlisten] connection accepted from 127.0.0.5:55893 #86 (8 connections now open) m31002| Fri Jan 18 13:11:56.396 [conn86] end connection 127.0.0.5:55893 (6 connections now open) m31003| Fri Jan 18 13:11:56.399 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:56.399 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:56.399 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:56.481 [initandlisten] connection accepted from 127.0.0.5:36028 #87 (7 connections now open) m31002| Fri Jan 18 13:11:56.627 [conn87] end connection 127.0.0.5:36028 (6 connections now open) m31003| Fri Jan 18 13:11:56.631 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:56.631 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:56.631 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:56.631 [initandlisten] connection accepted from 127.0.0.5:60445 #88 (7 connections now open) m31003| Fri Jan 18 13:11:56.808 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:56.808 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:56.808 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:56.862 [conn88] end connection 127.0.0.5:60445 (6 connections now open) m31002| Fri Jan 18 13:11:56.914 [initandlisten] connection accepted from 127.0.0.5:54999 #89 (7 connections now open) m31003| Fri Jan 18 13:11:57.003 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:57.003 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:57.003 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.003 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.003 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Fri Jan 18 13:11:57.076 [conn89] end connection 127.0.0.5:54999 (6 connections now open) m31001| Fri Jan 18 13:11:57.166 [initandlisten] connection accepted from 127.0.0.5:33070 #137 (6 connections now open) m31003| Fri Jan 18 13:11:57.250 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:57.250 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.250 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:57.250 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.250 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:57.307 [initandlisten] connection accepted from 127.0.0.5:36078 #138 (7 connections now open) m31001| Fri Jan 18 13:11:57.307 [conn137] end connection 127.0.0.5:33070 (6 connections now open) m31002| Fri Jan 18 13:11:57.340 [conn57] end connection 127.0.0.5:38685 (5 connections now open) m31001| Fri Jan 18 13:11:57.393 [initandlisten] connection accepted from 127.0.0.5:44938 #139 (7 connections now open) m31003| Fri Jan 18 13:11:57.461 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:57.461 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:57.461 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.461 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:57.515 [initandlisten] connection accepted from 127.0.0.5:45519 #140 (8 connections now open) m31001| Fri Jan 18 13:11:57.515 [conn138] end connection 127.0.0.5:36078 (7 connections now open) m31001| Fri Jan 18 13:11:57.646 [conn139] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532589000|5064 } } cursorid:876632960986992 ntoreturn:0 keyUpdates:0 numYields: 189 locks(micros) r:249836 nreturned:159641 reslen:2713917 131ms m31003| Fri Jan 18 13:11:57.659 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:57.659 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:57.659 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.659 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:57.664 [conn140] end connection 127.0.0.5:45519 (6 connections now open) m31001| Fri Jan 18 13:11:57.743 [initandlisten] connection accepted from 127.0.0.5:52560 #141 (7 connections now open) m31003| Fri Jan 18 13:11:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:57.797 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:57.868 [conn141] end connection 127.0.0.5:52560 (6 connections now open) m31001| Fri Jan 18 13:11:57.886 [initandlisten] connection accepted from 127.0.0.5:50726 #142 (7 connections now open) m31003| Fri Jan 18 13:11:58.085 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:58.085 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:58.085 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:58.085 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:58.143 [initandlisten] connection accepted from 127.0.0.5:41611 #143 (8 connections now open) m31001| Fri Jan 18 13:11:58.143 [conn142] end connection 127.0.0.5:50726 (7 connections now open) m31003| Fri Jan 18 13:11:58.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:58.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:58.290 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:58.290 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:58.352 [conn143] end connection 127.0.0.5:41611 (6 connections now open) m31001| Fri Jan 18 13:11:58.389 [initandlisten] connection accepted from 127.0.0.5:43508 #144 (7 connections now open) m31003| Fri Jan 18 13:11:58.490 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:58.490 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:58.490 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:58.490 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:58.561 [conn144] end connection 127.0.0.5:43508 (6 connections now open) m31001| Fri Jan 18 13:11:58.583 [initandlisten] connection accepted from 127.0.0.5:59730 #145 (7 connections now open) m31003| Fri Jan 18 13:11:58.817 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:58.817 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:58.817 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:11:58.817 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:11:58.911 [initandlisten] connection accepted from 127.0.0.5:39523 #146 (8 connections now open) m31001| Fri Jan 18 13:11:58.911 [conn145] end connection 127.0.0.5:59730 (6 connections now open) m31003| Fri Jan 18 13:11:59.041 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.041 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.041 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.041 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.066 [initandlisten] connection accepted from 127.0.0.5:36201 #90 (6 connections now open) m31001| Fri Jan 18 13:11:59.131 [conn146] end connection 127.0.0.5:39523 (6 connections now open) m31003| Fri Jan 18 13:11:59.208 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.208 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.208 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.208 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.289 [conn90] end connection 127.0.0.5:36201 (5 connections now open) m31001| Fri Jan 18 13:11:59.291 [conn139] end connection 127.0.0.5:44938 (5 connections now open) m31002| Fri Jan 18 13:11:59.306 [initandlisten] connection accepted from 127.0.0.5:36393 #91 (6 connections now open) m31002| Fri Jan 18 13:11:59.306 [initandlisten] connection accepted from 127.0.0.5:40650 #92 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 209, "optime" : { "t" : 1358532719000, "i" : 2728 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532718000, "i" : 3843 }, "optimeDate" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 174, "optime" : { "t" : 1358532717000, "i" : 3577 }, "optimeDate" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 209, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532589000, "i" : 5064 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532717000, "i" : 3562 }, "optimeDate" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 173, "optime" : { "t" : 1358532718000, "i" : 993 }, "optimeDate" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 173, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 208, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 173, "optime" : { "t" : 1358532589000, "i" : 5464 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 173, "optime" : { "t" : 1358532718000, "i" : 993 }, "optimeDate" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:59Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 19 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532589000, "i" : 5964 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 175, "optime" : { "t" : 1358532718000, "i" : 3576 }, "optimeDate" : ISODate("2013-01-18T18:11:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:11:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 174, "optime" : { "t" : 1358532717000, "i" : 3632 }, "optimeDate" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 174, "optime" : { "t" : 1358532589000, "i" : 5064 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:11:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532719000, "i" : 2721 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:11:59.381 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.381 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.381 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.385 [conn91] end connection 127.0.0.5:36393 (6 connections now open) m31002| Fri Jan 18 13:11:59.403 [initandlisten] connection accepted from 127.0.0.5:49969 #93 (7 connections now open) m31003| Fri Jan 18 13:11:59.474 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.474 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.474 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.521 [conn93] end connection 127.0.0.5:49969 (6 connections now open) m31002| Fri Jan 18 13:11:59.576 [initandlisten] connection accepted from 127.0.0.5:43995 #94 (7 connections now open) m31002| Fri Jan 18 13:11:59.583 [conn92] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532589000|5964 } } cursorid:882311930231357 ntoreturn:0 keyUpdates:0 locks(micros) r:126306 nreturned:158741 reslen:2698617 126ms m31003| Fri Jan 18 13:11:59.715 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.715 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.715 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.773 [initandlisten] connection accepted from 127.0.0.5:51278 #95 (8 connections now open) m31002| Fri Jan 18 13:11:59.773 [conn94] end connection 127.0.0.5:43995 (7 connections now open) m31003| Fri Jan 18 13:11:59.847 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:11:59.847 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:11:59.847 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:11:59.864 [initandlisten] connection accepted from 127.0.0.5:39654 #96 (8 connections now open) m31002| Fri Jan 18 13:11:59.941 [conn95] end connection 127.0.0.5:51278 (7 connections now open) m31003| Fri Jan 18 13:12:00.012 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.012 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.012 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:00.018 [conn96] end connection 127.0.0.5:39654 (6 connections now open) m31002| Fri Jan 18 13:12:00.065 [initandlisten] connection accepted from 127.0.0.5:59680 #97 (7 connections now open) m31002| Fri Jan 18 13:12:00.179 [conn97] end connection 127.0.0.5:59680 (6 connections now open) m31003| Fri Jan 18 13:12:00.183 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.183 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.183 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:00.258 [initandlisten] connection accepted from 127.0.0.5:44125 #98 (7 connections now open) m31003| Fri Jan 18 13:12:00.336 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.336 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.336 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31000| Fri Jan 18 13:12:00.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Fri Jan 18 13:12:00.417 [conn98] end connection 127.0.0.5:44125 (6 connections now open) m31002| Fri Jan 18 13:12:00.432 [initandlisten] connection accepted from 127.0.0.5:56880 #99 (7 connections now open) m31002| Fri Jan 18 13:12:00.525 [conn99] end connection 127.0.0.5:56880 (6 connections now open) m31003| Fri Jan 18 13:12:00.530 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.530 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.530 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:00.609 [initandlisten] connection accepted from 127.0.0.5:58845 #100 (7 connections now open) m31003| Fri Jan 18 13:12:00.670 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.670 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.670 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:00.711 [conn100] end connection 127.0.0.5:58845 (6 connections now open) m31002| Fri Jan 18 13:12:00.711 [initandlisten] connection accepted from 127.0.0.5:56343 #101 (8 connections now open) m31003| Fri Jan 18 13:12:00.766 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.766 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.766 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:00.815 [conn101] end connection 127.0.0.5:56343 (6 connections now open) m31002| Fri Jan 18 13:12:00.894 [initandlisten] connection accepted from 127.0.0.5:46033 #102 (7 connections now open) m31003| Fri Jan 18 13:12:00.949 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:00.949 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:00.949 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:01.033 [conn102] end connection 127.0.0.5:46033 (6 connections now open) m31002| Fri Jan 18 13:12:01.033 [initandlisten] connection accepted from 127.0.0.5:51508 #103 (8 connections now open) m31003| Fri Jan 18 13:12:01.142 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:01.143 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:01.143 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:01.191 [conn103] end connection 127.0.0.5:51508 (6 connections now open) m31002| Fri Jan 18 13:12:01.196 [initandlisten] connection accepted from 127.0.0.5:44660 #104 (7 connections now open) m31002| Fri Jan 18 13:12:01.266 [conn104] end connection 127.0.0.5:44660 (6 connections now open) m31003| Fri Jan 18 13:12:01.293 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:01.293 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:01.293 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:01.407 [initandlisten] connection accepted from 127.0.0.5:49321 #105 (7 connections now open) m31003| Fri Jan 18 13:12:01.551 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:01.551 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:01.551 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:01.628 [conn105] end connection 127.0.0.5:49321 (6 connections now open) m31002| Fri Jan 18 13:12:01.647 [initandlisten] connection accepted from 127.0.0.5:44024 #106 (7 connections now open) m31003| Fri Jan 18 13:12:01.800 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:01.800 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:01.800 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:01.871 [conn106] end connection 127.0.0.5:44024 (6 connections now open) m31002| Fri Jan 18 13:12:01.968 [initandlisten] connection accepted from 127.0.0.5:49165 #107 (7 connections now open) m31002| Fri Jan 18 13:12:02.082 [conn107] end connection 127.0.0.5:49165 (6 connections now open) m31003| Fri Jan 18 13:12:02.086 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:02.086 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:02.086 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:02.143 [initandlisten] connection accepted from 127.0.0.5:39304 #108 (7 connections now open) m31003| Fri Jan 18 13:12:02.255 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:02.255 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:02.255 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:02.347 [conn108] end connection 127.0.0.5:39304 (6 connections now open) m31002| Fri Jan 18 13:12:02.372 [initandlisten] connection accepted from 127.0.0.5:38171 #109 (7 connections now open) m31003| Fri Jan 18 13:12:02.488 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:02.488 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:02.488 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:02.499 [conn109] end connection 127.0.0.5:38171 (6 connections now open) m31002| Fri Jan 18 13:12:02.583 [initandlisten] connection accepted from 127.0.0.5:41349 #110 (7 connections now open) m31003| Fri Jan 18 13:12:02.739 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:02.739 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:02.739 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:02.806 [initandlisten] connection accepted from 127.0.0.5:36455 #111 (8 connections now open) m31002| Fri Jan 18 13:12:02.806 [conn110] end connection 127.0.0.5:41349 (6 connections now open) m31003| Fri Jan 18 13:12:02.892 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:02.892 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:02.892 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:02.983 [conn111] end connection 127.0.0.5:36455 (6 connections now open) m31002| Fri Jan 18 13:12:03.006 [initandlisten] connection accepted from 127.0.0.5:55864 #112 (7 connections now open) m31003| Fri Jan 18 13:12:03.105 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.105 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.105 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:03.148 [conn112] end connection 127.0.0.5:55864 (6 connections now open) m31002| Fri Jan 18 13:12:03.208 [initandlisten] connection accepted from 127.0.0.5:38020 #113 (7 connections now open) m31002| Fri Jan 18 13:12:03.311 [conn113] end connection 127.0.0.5:38020 (6 connections now open) m31003| Fri Jan 18 13:12:03.313 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.313 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.313 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:03.328 [initandlisten] connection accepted from 127.0.0.5:48354 #114 (7 connections now open) m31003| Fri Jan 18 13:12:03.455 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.455 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.455 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:03.518 [conn114] end connection 127.0.0.5:48354 (6 connections now open) m31002| Fri Jan 18 13:12:03.537 [initandlisten] connection accepted from 127.0.0.5:51280 #115 (7 connections now open) m31003| Fri Jan 18 13:12:03.582 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.582 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.582 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:03.628 [conn115] end connection 127.0.0.5:51280 (6 connections now open) m31002| Fri Jan 18 13:12:03.637 [initandlisten] connection accepted from 127.0.0.5:56778 #116 (7 connections now open) m31003| Fri Jan 18 13:12:03.745 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.745 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.745 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:03.810 [conn116] end connection 127.0.0.5:56778 (6 connections now open) m31002| Fri Jan 18 13:12:03.859 [initandlisten] connection accepted from 127.0.0.5:36027 #117 (7 connections now open) m31003| Fri Jan 18 13:12:03.936 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:03.936 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:03.936 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:04.026 [conn117] end connection 127.0.0.5:36027 (6 connections now open) m31002| Fri Jan 18 13:12:04.036 [initandlisten] connection accepted from 127.0.0.5:44637 #118 (7 connections now open) m31003| Fri Jan 18 13:12:04.166 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:04.166 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:04.166 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:04.249 [conn118] end connection 127.0.0.5:44637 (6 connections now open) m31002| Fri Jan 18 13:12:04.270 [initandlisten] connection accepted from 127.0.0.5:48082 #119 (7 connections now open) m31003| Fri Jan 18 13:12:04.401 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:04.401 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:04.401 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:04.486 [conn119] end connection 127.0.0.5:48082 (6 connections now open) m31002| Fri Jan 18 13:12:04.486 [initandlisten] connection accepted from 127.0.0.5:54718 #120 (8 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 214, "optime" : { "t" : 1358532724000, "i" : 4100 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532724000, "i" : 3514 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 179, "optime" : { "t" : 1358532723000, "i" : 1338 }, "optimeDate" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 214, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532589000, "i" : 8164 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532723000, "i" : 1338 }, "optimeDate" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:03Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 178, "optime" : { "t" : 1358532724000, "i" : 635 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 178, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 178, "optime" : { "t" : 1358532589000, "i" : 8564 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 178, "optime" : { "t" : 1358532724000, "i" : 585 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:04Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:03Z"), "pingMs" : 26 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 25 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532589000, "i" : 8764 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 180, "optime" : { "t" : 1358532723000, "i" : 1338 }, "optimeDate" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:03Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 179, "optime" : { "t" : 1358532723000, "i" : 2102 }, "optimeDate" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:04Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 179, "optime" : { "t" : 1358532589000, "i" : 8164 }, "optimeDate" : ISODate("2013-01-18T18:09:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532724000, "i" : 4100 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:12:04.605 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:04.605 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:04.605 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:04.633 [conn120] end connection 127.0.0.5:54718 (6 connections now open) m31002| Fri Jan 18 13:12:04.721 [initandlisten] connection accepted from 127.0.0.5:35216 #121 (7 connections now open) m31003| Fri Jan 18 13:12:04.873 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:04.873 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:04.873 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:04.949 [conn121] end connection 127.0.0.5:35216 (6 connections now open) m31002| Fri Jan 18 13:12:04.949 [initandlisten] connection accepted from 127.0.0.5:52187 #122 (8 connections now open) m31002| Fri Jan 18 13:12:05.049 [conn122] end connection 127.0.0.5:52187 (6 connections now open) m31003| Fri Jan 18 13:12:05.053 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:05.053 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:05.053 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.053 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.053 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:12:05.053 [initandlisten] connection accepted from 127.0.0.5:58273 #147 (6 connections now open) m31003| Fri Jan 18 13:12:05.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:05.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:05.241 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.241 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:05.312 [conn147] end connection 127.0.0.5:58273 (5 connections now open) m31001| Fri Jan 18 13:12:05.325 [initandlisten] connection accepted from 127.0.0.5:59609 #148 (6 connections now open) m31001| Fri Jan 18 13:12:05.325 [initandlisten] connection accepted from 127.0.0.5:48195 #149 (7 connections now open) m31002| Fri Jan 18 13:12:05.340 [conn92] end connection 127.0.0.5:40650 (5 connections now open) m31003| Fri Jan 18 13:12:05.395 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:05.395 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:05.395 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.395 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:05.425 [conn149] end connection 127.0.0.5:48195 (6 connections now open) m31001| Fri Jan 18 13:12:05.490 [initandlisten] connection accepted from 127.0.0.5:35399 #150 (7 connections now open) m31001| Fri Jan 18 13:12:05.550 [conn148] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532590000|305 } } cursorid:910443096651797 ntoreturn:0 keyUpdates:0 locks(micros) r:125114 nreturned:155541 reslen:2644217 125ms m31003| Fri Jan 18 13:12:05.634 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:05.634 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:05.634 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.634 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:05.699 [initandlisten] connection accepted from 127.0.0.5:43485 #151 (8 connections now open) m31001| Fri Jan 18 13:12:05.699 [conn150] end connection 127.0.0.5:35399 (7 connections now open) m31003| Fri Jan 18 13:12:05.830 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:05.830 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:05.830 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:05.830 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:05.844 [initandlisten] connection accepted from 127.0.0.5:47083 #152 (8 connections now open) m31001| Fri Jan 18 13:12:05.864 [conn151] end connection 127.0.0.5:43485 (7 connections now open) m31004| Fri Jan 18 13:12:05.879 [conn25] end connection 127.0.0.5:44848 (4 connections now open) m31001| Fri Jan 18 13:12:05.993 [conn152] end connection 127.0.0.5:47083 (6 connections now open) m31003| Fri Jan 18 13:12:06.025 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:06.025 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:06.025 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:06.025 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:06.093 [initandlisten] connection accepted from 127.0.0.5:36514 #153 (7 connections now open) m31003| Fri Jan 18 13:12:06.231 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:06.231 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:06.231 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:06.231 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:06.245 [conn153] end connection 127.0.0.5:36514 (6 connections now open) m31001| Fri Jan 18 13:12:06.327 [initandlisten] connection accepted from 127.0.0.5:43171 #154 (7 connections now open) m31004| Fri Jan 18 13:12:06.380 [initandlisten] connection accepted from 127.0.0.5:41527 #29 (5 connections now open) m31004| Fri Jan 18 13:12:06.442 [conn26] end connection 127.0.0.2:33659 (4 connections now open) m31004| Fri Jan 18 13:12:06.442 [initandlisten] connection accepted from 127.0.0.2:49811 #30 (5 connections now open) m31003| Fri Jan 18 13:12:06.444 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:06.444 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:06.444 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:06.444 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:06.498 [conn154] end connection 127.0.0.5:43171 (6 connections now open) m31001| Fri Jan 18 13:12:06.570 [initandlisten] connection accepted from 127.0.0.5:58475 #155 (7 connections now open) m31003| Fri Jan 18 13:12:06.687 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:06.687 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:06.687 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:06.687 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:06.733 [conn155] end connection 127.0.0.5:58475 (6 connections now open) m31001| Fri Jan 18 13:12:06.744 [initandlisten] connection accepted from 127.0.0.5:48352 #156 (7 connections now open) m31001| Fri Jan 18 13:12:06.833 [conn156] end connection 127.0.0.5:48352 (6 connections now open) m31003| Fri Jan 18 13:12:06.837 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:06.837 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:06.837 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:06.837 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:06.961 [initandlisten] connection accepted from 127.0.0.5:34579 #157 (7 connections now open) m31003| Fri Jan 18 13:12:07.081 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:07.081 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:07.081 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:07.081 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:07.170 [conn157] end connection 127.0.0.5:34579 (6 connections now open) m31001| Fri Jan 18 13:12:07.170 [initandlisten] connection accepted from 127.0.0.5:48644 #158 (8 connections now open) m31003| Fri Jan 18 13:12:07.318 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:07.318 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:07.318 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:07.318 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:07.395 [conn158] end connection 127.0.0.5:48644 (6 connections now open) m31004| Fri Jan 18 13:12:07.397 [conn27] end connection 127.0.0.3:36333 (4 connections now open) m31004| Fri Jan 18 13:12:07.397 [initandlisten] connection accepted from 127.0.0.3:39519 #31 (5 connections now open) m31003| Fri Jan 18 13:12:07.445 [conn25] end connection 127.0.0.6:47817 (3 connections now open) m31003| Fri Jan 18 13:12:07.445 [initandlisten] connection accepted from 127.0.0.6:34386 #28 (5 connections now open) m31001| Fri Jan 18 13:12:07.471 [initandlisten] connection accepted from 127.0.0.5:59973 #159 (7 connections now open) m31003| Fri Jan 18 13:12:07.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:07.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:07.524 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:07.524 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:07.587 [conn159] end connection 127.0.0.5:59973 (6 connections now open) m31001| Fri Jan 18 13:12:07.587 [initandlisten] connection accepted from 127.0.0.5:34603 #160 (8 connections now open) m31001| Fri Jan 18 13:12:07.651 [conn160] end connection 127.0.0.5:34603 (6 connections now open) m31003| Fri Jan 18 13:12:07.654 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:07.654 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:07.654 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:07.654 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:07.746 [initandlisten] connection accepted from 127.0.0.5:47379 #161 (7 connections now open) m31001| Fri Jan 18 13:12:07.878 [conn161] end connection 127.0.0.5:47379 (6 connections now open) m31003| Fri Jan 18 13:12:07.881 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:07.881 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:07.881 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:07.881 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:07.962 [initandlisten] connection accepted from 127.0.0.5:48830 #162 (7 connections now open) m31003| Fri Jan 18 13:12:08.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:08.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:08.058 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:08.058 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31004| Fri Jan 18 13:12:08.095 [conn28] end connection 127.0.0.4:49903 (4 connections now open) m31004| Fri Jan 18 13:12:08.095 [initandlisten] connection accepted from 127.0.0.4:48460 #32 (5 connections now open) m31001| Fri Jan 18 13:12:08.106 [conn162] end connection 127.0.0.5:48830 (6 connections now open) m31001| Fri Jan 18 13:12:08.118 [initandlisten] connection accepted from 127.0.0.5:50869 #163 (7 connections now open) m31003| Fri Jan 18 13:12:08.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:08.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:08.222 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:08.222 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:08.310 [initandlisten] connection accepted from 127.0.0.5:38197 #164 (8 connections now open) m31001| Fri Jan 18 13:12:08.310 [conn163] end connection 127.0.0.5:50869 (7 connections now open) m31003| Fri Jan 18 13:12:08.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:08.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:08.416 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:08.416 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:12:08.442 [conn30] end connection 127.0.0.2:56843 (4 connections now open) m31002| Fri Jan 18 13:12:08.442 [initandlisten] connection accepted from 127.0.0.2:37295 #123 (5 connections now open) m31001| Fri Jan 18 13:12:08.508 [conn164] end connection 127.0.0.5:38197 (6 connections now open) m31001| Fri Jan 18 13:12:08.603 [initandlisten] connection accepted from 127.0.0.5:49756 #165 (7 connections now open) m31003| Fri Jan 18 13:12:08.732 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:08.732 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:08.732 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:08.732 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:08.795 [initandlisten] connection accepted from 127.0.0.5:57583 #166 (8 connections now open) m31001| Fri Jan 18 13:12:08.795 [conn165] end connection 127.0.0.5:49756 (6 connections now open) m31001| Fri Jan 18 13:12:08.877 [conn166] end connection 127.0.0.5:57583 (6 connections now open) m31003| Fri Jan 18 13:12:08.879 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:08.879 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:08.879 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:08.879 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:08.902 [initandlisten] connection accepted from 127.0.0.5:44617 #167 (7 connections now open) m31003| Fri Jan 18 13:12:09.013 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.013 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.013 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.013 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.109 [initandlisten] connection accepted from 127.0.0.5:52749 #168 (8 connections now open) m31001| Fri Jan 18 13:12:09.109 [conn167] end connection 127.0.0.5:44617 (6 connections now open) m31003| Fri Jan 18 13:12:09.171 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.171 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.171 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.171 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.207 [conn168] end connection 127.0.0.5:52749 (6 connections now open) m31001| Fri Jan 18 13:12:09.219 [initandlisten] connection accepted from 127.0.0.5:33438 #169 (7 connections now open) m31003| Fri Jan 18 13:12:09.305 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.305 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.305 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.305 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.329 [conn169] end connection 127.0.0.5:33438 (6 connections now open) m31001| Fri Jan 18 13:12:09.350 [initandlisten] connection accepted from 127.0.0.5:34088 #170 (7 connections now open) m31002| Fri Jan 18 13:12:09.397 [conn31] end connection 127.0.0.3:42792 (4 connections now open) m31002| Fri Jan 18 13:12:09.397 [initandlisten] connection accepted from 127.0.0.3:58882 #124 (6 connections now open) m31001| Fri Jan 18 13:12:09.445 [conn90] end connection 127.0.0.6:38465 (6 connections now open) m31001| Fri Jan 18 13:12:09.445 [initandlisten] connection accepted from 127.0.0.6:52171 #171 (7 connections now open) m31001| Fri Jan 18 13:12:09.478 [conn170] end connection 127.0.0.5:34088 (6 connections now open) m31003| Fri Jan 18 13:12:09.482 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.482 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.482 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.482 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.492 [initandlisten] connection accepted from 127.0.0.5:55537 #172 (7 connections now open) m31003| Fri Jan 18 13:12:09.565 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.565 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.565 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.565 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.567 [conn172] end connection 127.0.0.5:55537 (6 connections now open) m31001| Fri Jan 18 13:12:09.664 [initandlisten] connection accepted from 127.0.0.5:43534 #173 (7 connections now open) m31001| Fri Jan 18 13:12:09.761 [conn173] end connection 127.0.0.5:43534 (6 connections now open) m31003| Fri Jan 18 13:12:09.763 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.763 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.763 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.763 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 219, "optime" : { "t" : 1358532729000, "i" : 7208 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532728000, "i" : 3335 }, "optimeDate" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.818 [initandlisten] connection accepted from 127.0.0.5:34875 #174 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 184, "optime" : { "t" : 1358532729000, "i" : 3710 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 219, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532593000, "i" : 1600 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532729000, "i" : 3573 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 183, "optime" : { "t" : 1358532728000, "i" : 473 }, "optimeDate" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 183, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 183, "optime" : { "t" : 1358532593000, "i" : 900 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 183, "optime" : { "t" : 1358532728000, "i" : 473 }, "optimeDate" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:09Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 14 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 14 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532593000, "i" : 1900 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 185, "optime" : { "t" : 1358532728000, "i" : 7640 }, "optimeDate" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:09Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 184, "optime" : { "t" : 1358532729000, "i" : 4163 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532661000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 184, "optime" : { "t" : 1358532593000, "i" : 1600 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532729000, "i" : 7208 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "self" : true } ], "ok" : 1 } m31003| Fri Jan 18 13:12:09.957 [conn1] command admin.$cmd command: { replSetGetStatus: true } ntoreturn:1 keyUpdates:0 reslen:1149 130ms { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:12:09.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:09.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:09.961 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:09.961 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:09.991 [conn174] end connection 127.0.0.5:34875 (6 connections now open) m31001| Fri Jan 18 13:12:10.006 [initandlisten] connection accepted from 127.0.0.5:50570 #175 (7 connections now open) m31001| Fri Jan 18 13:12:10.095 [conn95] end connection 127.0.0.4:42201 (6 connections now open) m31001| Fri Jan 18 13:12:10.095 [initandlisten] connection accepted from 127.0.0.4:52897 #176 (7 connections now open) m31003| Fri Jan 18 13:12:10.138 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:10.138 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:10.138 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:10.138 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:10.207 [conn175] end connection 127.0.0.5:50570 (6 connections now open) m31001| Fri Jan 18 13:12:10.239 [initandlisten] connection accepted from 127.0.0.5:49195 #177 (7 connections now open) m31003| Fri Jan 18 13:12:10.344 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:10.344 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:10.344 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:10.344 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:10.409 [conn177] end connection 127.0.0.5:49195 (6 connections now open) m31000| Fri Jan 18 13:12:10.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Fri Jan 18 13:12:10.485 [initandlisten] connection accepted from 127.0.0.5:52516 #178 (7 connections now open) m31001| Fri Jan 18 13:12:10.686 [conn178] end connection 127.0.0.5:52516 (6 connections now open) m31001| Fri Jan 18 13:12:10.967 [conn98] end connection 127.0.0.5:44984 (5 connections now open) m31003| Fri Jan 18 13:12:11.038 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:11.038 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.038 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:11.038 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:11.121 [initandlisten] connection accepted from 127.0.0.5:51342 #179 (6 connections now open) m31001| Fri Jan 18 13:12:11.121 [initandlisten] connection accepted from 127.0.0.5:44651 #180 (7 connections now open) m31000| Fri Jan 18 13:12:11.211 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 numYields: 61 locks(micros) r:259538 nreturned:160000 reslen:2720020 138ms m31003| Fri Jan 18 13:12:11.220 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:11.220 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.220 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.220 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31000| Fri Jan 18 13:12:11.229 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 numYields: 8 locks(micros) r:200094 nreturned:160000 reslen:2720020 156ms m31001| Fri Jan 18 13:12:11.275 [conn179] end connection 127.0.0.5:51342 (6 connections now open) m31002| Fri Jan 18 13:12:11.391 [initandlisten] connection accepted from 127.0.0.5:56166 #125 (6 connections now open) m31003| Fri Jan 18 13:12:11.471 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Fri Jan 18 13:12:11.513 [conn148] end connection 127.0.0.5:59609 (5 connections now open) m31002| Fri Jan 18 13:12:11.521 [initandlisten] connection accepted from 127.0.0.5:47098 #126 (7 connections now open) m31002| Fri Jan 18 13:12:11.521 [conn125] end connection 127.0.0.5:56166 (6 connections now open) m31003| Fri Jan 18 13:12:11.522 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:11.522 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.522 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:11.605 [initandlisten] connection accepted from 127.0.0.5:51205 #127 (7 connections now open) m31003| Fri Jan 18 13:12:11.659 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:11.659 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.659 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:11.677 [conn127] end connection 127.0.0.5:51205 (6 connections now open) m31002| Fri Jan 18 13:12:11.695 [initandlisten] connection accepted from 127.0.0.5:39869 #128 (7 connections now open) m31002| Fri Jan 18 13:12:11.831 [conn126] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532593000|2500 } } cursorid:934122265620565 ntoreturn:0 keyUpdates:0 locks(micros) r:135292 nreturned:167403 reslen:2845871 135ms m31003| Fri Jan 18 13:12:11.907 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:11.907 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:11.907 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:11.934 [conn128] end connection 127.0.0.5:39869 (6 connections now open) m31002| Fri Jan 18 13:12:11.934 [initandlisten] connection accepted from 127.0.0.5:52442 #129 (8 connections now open) m31003| Fri Jan 18 13:12:12.034 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:12.034 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:12.034 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:12.053 [initandlisten] connection accepted from 127.0.0.5:46806 #130 (8 connections now open) m31002| Fri Jan 18 13:12:12.083 [conn129] end connection 127.0.0.5:52442 (7 connections now open) m31003| Fri Jan 18 13:12:12.169 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:12.169 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:12.169 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:12.191 [initandlisten] connection accepted from 127.0.0.5:54296 #131 (8 connections now open) m31002| Fri Jan 18 13:12:12.252 [conn130] end connection 127.0.0.5:46806 (7 connections now open) m31002| Fri Jan 18 13:12:12.265 [conn131] end connection 127.0.0.5:54296 (6 connections now open) m31003| Fri Jan 18 13:12:12.268 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:12.268 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:12.268 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:12.310 [initandlisten] connection accepted from 127.0.0.5:50512 #132 (7 connections now open) m31002| Fri Jan 18 13:12:12.491 [conn132] end connection 127.0.0.5:50512 (6 connections now open) m31003| Fri Jan 18 13:12:12.494 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:12.494 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:12.494 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:12.531 [initandlisten] connection accepted from 127.0.0.5:58346 #133 (7 connections now open) m31003| Fri Jan 18 13:12:12.787 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:12.787 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:12.787 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:12.801 [conn133] end connection 127.0.0.5:58346 (6 connections now open) m31002| Fri Jan 18 13:12:12.929 [initandlisten] connection accepted from 127.0.0.5:38876 #134 (7 connections now open) m31003| Fri Jan 18 13:12:13.071 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.071 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.071 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.101 [initandlisten] connection accepted from 127.0.0.5:43454 #135 (8 connections now open) m31002| Fri Jan 18 13:12:13.101 [conn134] end connection 127.0.0.5:38876 (7 connections now open) m31003| Fri Jan 18 13:12:13.183 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.183 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.183 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.206 [initandlisten] connection accepted from 127.0.0.5:57549 #136 (8 connections now open) m31002| Fri Jan 18 13:12:13.206 [conn135] end connection 127.0.0.5:43454 (6 connections now open) m31002| Fri Jan 18 13:12:13.274 [conn136] end connection 127.0.0.5:57549 (6 connections now open) m31003| Fri Jan 18 13:12:13.277 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.277 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.277 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.362 [initandlisten] connection accepted from 127.0.0.5:41851 #137 (7 connections now open) m31003| Fri Jan 18 13:12:13.498 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.498 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.498 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.590 [initandlisten] connection accepted from 127.0.0.5:50335 #138 (8 connections now open) m31002| Fri Jan 18 13:12:13.590 [conn137] end connection 127.0.0.5:41851 (6 connections now open) m31003| Fri Jan 18 13:12:13.758 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.758 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.758 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.820 [initandlisten] connection accepted from 127.0.0.5:50013 #139 (8 connections now open) m31002| Fri Jan 18 13:12:13.820 [conn138] end connection 127.0.0.5:50335 (6 connections now open) m31003| Fri Jan 18 13:12:13.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Fri Jan 18 13:12:13.939 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:13.939 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:13.939 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:13.993 [conn139] end connection 127.0.0.5:50013 (6 connections now open) m31002| Fri Jan 18 13:12:14.022 [initandlisten] connection accepted from 127.0.0.5:53477 #140 (7 connections now open) m31003| Fri Jan 18 13:12:14.136 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:14.136 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:14.136 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:14.184 [conn140] end connection 127.0.0.5:53477 (6 connections now open) m31002| Fri Jan 18 13:12:14.236 [initandlisten] connection accepted from 127.0.0.5:49807 #141 (7 connections now open) m31003| Fri Jan 18 13:12:14.287 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:14.287 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:14.287 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:14.342 [conn141] end connection 127.0.0.5:49807 (6 connections now open) m31002| Fri Jan 18 13:12:14.354 [initandlisten] connection accepted from 127.0.0.5:37798 #142 (7 connections now open) m31002| Fri Jan 18 13:12:14.449 [conn142] end connection 127.0.0.5:37798 (6 connections now open) m31003| Fri Jan 18 13:12:14.451 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:14.451 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:14.451 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:14.566 [initandlisten] connection accepted from 127.0.0.5:43179 #143 (7 connections now open) m31002| Fri Jan 18 13:12:14.675 [conn143] end connection 127.0.0.5:43179 (6 connections now open) m31003| Fri Jan 18 13:12:14.677 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:14.677 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:14.677 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:14.759 [initandlisten] connection accepted from 127.0.0.5:48508 #144 (7 connections now open) m31002| Fri Jan 18 13:12:14.951 [conn144] end connection 127.0.0.5:48508 (6 connections now open) m31003| Fri Jan 18 13:12:14.954 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:14.954 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:14.954 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:15.074 [initandlisten] connection accepted from 127.0.0.5:45191 #145 (7 connections now open) m31002| Fri Jan 18 13:12:15.168 [conn145] end connection 127.0.0.5:45191 (6 connections now open) STATUS FROM : 127.0.0.2:31000 m31003| Fri Jan 18 13:12:15.171 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:15.171 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:15.171 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 225, "optime" : { "t" : 1358532735000, "i" : 1782 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532734000, "i" : 3350 }, "optimeDate" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 190, "optime" : { "t" : 1358532733000, "i" : 4032 }, "optimeDate" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 225, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:14Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532593000, "i" : 3500 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532733000, "i" : 4014 }, "optimeDate" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 189, "optime" : { "t" : 1358532734000, "i" : 472 }, "optimeDate" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 189, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 224, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 189, "optime" : { "t" : 1358532593000, "i" : 3800 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:15Z"), "pingMs" : 227, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 189, "optime" : { "t" : 1358532734000, "i" : 466 }, "optimeDate" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:15Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 18 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 16 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532593000, "i" : 4400 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 191, "optime" : { "t" : 1358532733000, "i" : 8248 }, "optimeDate" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 190, "optime" : { "t" : 1358532733000, "i" : 4386 }, "optimeDate" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532692000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:14Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 190, "optime" : { "t" : 1358532593000, "i" : 3500 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532735000, "i" : 1782 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:12:15.266 [initandlisten] connection accepted from 127.0.0.5:35849 #146 (7 connections now open) m31003| Fri Jan 18 13:12:15.409 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:15.409 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:15.409 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:15.447 [conn146] end connection 127.0.0.5:35849 (6 connections now open) m31002| Fri Jan 18 13:12:15.448 [initandlisten] connection accepted from 127.0.0.5:41459 #147 (8 connections now open) m31003| Fri Jan 18 13:12:15.642 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:15.642 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:15.642 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:15.716 [initandlisten] connection accepted from 127.0.0.5:58926 #148 (8 connections now open) m31002| Fri Jan 18 13:12:15.716 [conn147] end connection 127.0.0.5:41459 (6 connections now open) m31002| Fri Jan 18 13:12:15.847 [conn148] end connection 127.0.0.5:58926 (6 connections now open) m31003| Fri Jan 18 13:12:15.851 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:15.851 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:15.851 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:15.915 [initandlisten] connection accepted from 127.0.0.5:44726 #149 (7 connections now open) m31003| Fri Jan 18 13:12:16.038 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:16.038 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:16.038 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:16.110 [initandlisten] connection accepted from 127.0.0.5:47938 #150 (8 connections now open) m31002| Fri Jan 18 13:12:16.110 [conn149] end connection 127.0.0.5:44726 (6 connections now open) m31002| Fri Jan 18 13:12:16.255 [conn150] end connection 127.0.0.5:47938 (6 connections now open) m31003| Fri Jan 18 13:12:16.258 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:16.258 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:16.258 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:16.310 [initandlisten] connection accepted from 127.0.0.5:45339 #151 (7 connections now open) m31003| Fri Jan 18 13:12:16.455 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:16.455 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:16.455 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:16.541 [conn151] end connection 127.0.0.5:45339 (6 connections now open) m31002| Fri Jan 18 13:12:16.541 [initandlisten] connection accepted from 127.0.0.5:59768 #152 (8 connections now open) m31003| Fri Jan 18 13:12:16.604 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:16.604 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:16.604 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:16.697 [initandlisten] connection accepted from 127.0.0.5:58800 #153 (8 connections now open) m31002| Fri Jan 18 13:12:16.697 [conn152] end connection 127.0.0.5:59768 (6 connections now open) m31003| Fri Jan 18 13:12:16.822 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:16.822 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:16.822 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:16.891 [initandlisten] connection accepted from 127.0.0.5:38571 #154 (8 connections now open) m31002| Fri Jan 18 13:12:16.891 [conn153] end connection 127.0.0.5:58800 (7 connections now open) m31002| Fri Jan 18 13:12:17.017 [conn154] end connection 127.0.0.5:38571 (6 connections now open) m31003| Fri Jan 18 13:12:17.021 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:17.021 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:17.021 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:17.038 [initandlisten] connection accepted from 127.0.0.5:35275 #155 (7 connections now open) m31003| Fri Jan 18 13:12:17.143 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:17.143 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:17.143 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.143 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.143 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:12:17.143 [initandlisten] connection accepted from 127.0.0.5:52483 #181 (6 connections now open) m31002| Fri Jan 18 13:12:17.207 [conn155] end connection 127.0.0.5:35275 (6 connections now open) m31003| Fri Jan 18 13:12:17.296 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:17.296 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.296 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:17.296 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.296 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:17.313 [conn181] end connection 127.0.0.5:52483 (5 connections now open) m31002| Fri Jan 18 13:12:17.328 [conn126] end connection 127.0.0.5:47098 (5 connections now open) m31001| Fri Jan 18 13:12:17.379 [initandlisten] connection accepted from 127.0.0.5:34641 #182 (6 connections now open) m31001| Fri Jan 18 13:12:17.460 [initandlisten] connection accepted from 127.0.0.5:48038 #183 (7 connections now open) m31003| Fri Jan 18 13:12:17.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:17.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:17.572 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.572 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:17.638 [conn183] end connection 127.0.0.5:48038 (6 connections now open) m31001| Fri Jan 18 13:12:17.658 [initandlisten] connection accepted from 127.0.0.5:37865 #184 (7 connections now open) m31001| Fri Jan 18 13:12:17.759 [conn182] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532593000|5500 } } cursorid:961983049530254 ntoreturn:0 keyUpdates:0 numYields: 1138 locks(micros) r:304725 nreturned:181336 reslen:3082732 154ms m31003| Fri Jan 18 13:12:17.818 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:17.818 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:17.818 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:17.818 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:17.830 [initandlisten] connection accepted from 127.0.0.5:38174 #185 (8 connections now open) m31001| Fri Jan 18 13:12:17.844 [conn184] end connection 127.0.0.5:37865 (7 connections now open) m31003| Fri Jan 18 13:12:18.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:18.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:18.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:18.167 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:18.191 [initandlisten] connection accepted from 127.0.0.5:47265 #186 (8 connections now open) m31001| Fri Jan 18 13:12:18.221 [conn185] end connection 127.0.0.5:38174 (7 connections now open) m31003| Fri Jan 18 13:12:18.272 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:18.272 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:18.272 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:18.272 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:18.319 [conn186] end connection 127.0.0.5:47265 (6 connections now open) m31001| Fri Jan 18 13:12:18.355 [initandlisten] connection accepted from 127.0.0.5:37631 #187 (7 connections now open) m31003| Fri Jan 18 13:12:18.514 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:18.514 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:18.514 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:18.514 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:18.561 [conn187] end connection 127.0.0.5:37631 (6 connections now open) m31001| Fri Jan 18 13:12:18.649 [initandlisten] connection accepted from 127.0.0.5:43958 #188 (7 connections now open) m31003| Fri Jan 18 13:12:18.718 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:18.718 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:18.719 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:18.719 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:18.790 [conn188] end connection 127.0.0.5:43958 (6 connections now open) m31001| Fri Jan 18 13:12:18.872 [initandlisten] connection accepted from 127.0.0.5:55520 #189 (7 connections now open) m31003| Fri Jan 18 13:12:18.925 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:18.925 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:18.925 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:18.925 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:18.946 [conn189] end connection 127.0.0.5:55520 (6 connections now open) m31001| Fri Jan 18 13:12:18.946 [initandlisten] connection accepted from 127.0.0.5:34533 #190 (8 connections now open) m31003| Fri Jan 18 13:12:19.064 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:19.064 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:19.064 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:19.064 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:19.148 [conn190] end connection 127.0.0.5:34533 (6 connections now open) m31001| Fri Jan 18 13:12:19.161 [initandlisten] connection accepted from 127.0.0.5:50564 #191 (7 connections now open) m31003| Fri Jan 18 13:12:19.392 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:19.392 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:19.392 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:19.392 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:19.433 [initandlisten] connection accepted from 127.0.0.5:52543 #156 (6 connections now open) m31001| Fri Jan 18 13:12:19.446 [conn191] end connection 127.0.0.5:50564 (6 connections now open) m31002| Fri Jan 18 13:12:19.581 [conn156] end connection 127.0.0.5:52543 (5 connections now open) m31003| Fri Jan 18 13:12:19.583 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:19.583 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:19.583 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:19.640 [initandlisten] connection accepted from 127.0.0.5:37550 #157 (6 connections now open) m31003| Fri Jan 18 13:12:19.754 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Fri Jan 18 13:12:19.755 [conn182] end connection 127.0.0.5:34641 (5 connections now open) m31002| Fri Jan 18 13:12:19.854 [initandlisten] connection accepted from 127.0.0.5:47423 #158 (7 connections now open) m31003| Fri Jan 18 13:12:19.857 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:19.857 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:19.857 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:19.913 [conn157] end connection 127.0.0.5:37550 (6 connections now open) m31002| Fri Jan 18 13:12:19.945 [initandlisten] connection accepted from 127.0.0.5:45616 #159 (7 connections now open) m31003| Fri Jan 18 13:12:20.062 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.062 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:20.062 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:20.089 [conn159] end connection 127.0.0.5:45616 (6 connections now open) m31002| Fri Jan 18 13:12:20.129 [conn158] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532593000|6500 } } cursorid:969470750131771 ntoreturn:0 keyUpdates:0 locks(micros) r:167005 nreturned:188592 reslen:3206084 166ms m31002| Fri Jan 18 13:12:20.196 [initandlisten] connection accepted from 127.0.0.5:35312 #160 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 230, "optime" : { "t" : 1358532740000, "i" : 3321 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:20Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:11:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532738000, "i" : 2037 }, "optimeDate" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 195, "optime" : { "t" : 1358532739000, "i" : 4326 }, "optimeDate" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 230, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:20Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532593000, "i" : 6400 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532739000, "i" : 4207 }, "optimeDate" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 194, "optime" : { "t" : 1358532740000, "i" : 1832 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 194, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 229, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 194, "optime" : { "t" : 1358532593000, "i" : 5900 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 144, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 194, "optime" : { "t" : 1358532740000, "i" : 1405 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:20Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 14 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 12 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532593000, "i" : 6700 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 196, "optime" : { "t" : 1358532738000, "i" : 6391 }, "optimeDate" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:19Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.339 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.339 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:20.339 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 195, "optime" : { "t" : 1358532739000, "i" : 4409 }, "optimeDate" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532697000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:20Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 195, "optime" : { "t" : 1358532593000, "i" : 6400 }, "optimeDate" : ISODate("2013-01-18T18:09:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532740000, "i" : 3321 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:12:20.428 [initandlisten] connection accepted from 127.0.0.5:42746 #161 (8 connections now open) m31002| Fri Jan 18 13:12:20.436 [conn160] end connection 127.0.0.5:35312 (7 connections now open) m31003| Fri Jan 18 13:12:20.495 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.495 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:20.495 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:20.526 [initandlisten] connection accepted from 127.0.0.5:50706 #162 (8 connections now open) m31002| Fri Jan 18 13:12:20.576 [conn161] end connection 127.0.0.5:42746 (7 connections now open) m31003| Fri Jan 18 13:12:20.663 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.663 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:20.663 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:20.675 [conn162] end connection 127.0.0.5:50706 (6 connections now open) m31002| Fri Jan 18 13:12:20.741 [initandlisten] connection accepted from 127.0.0.5:37603 #163 (7 connections now open) m31003| Fri Jan 18 13:12:20.914 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:20.914 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:20.914 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:20.996 [conn163] end connection 127.0.0.5:37603 (6 connections now open) m31002| Fri Jan 18 13:12:21.013 [initandlisten] connection accepted from 127.0.0.5:40599 #164 (7 connections now open) m31003| Fri Jan 18 13:12:21.142 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:21.142 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.142 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:21.142 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:21.142 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:12:21.220 [initandlisten] connection accepted from 127.0.0.5:45459 #192 (6 connections now open) m31002| Fri Jan 18 13:12:21.236 [conn164] end connection 127.0.0.5:40599 (6 connections now open) m31001| Fri Jan 18 13:12:21.374 [conn192] end connection 127.0.0.5:45459 (5 connections now open) m31003| Fri Jan 18 13:12:21.376 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:21.376 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.376 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.376 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:21.376 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:21.376 [conn158] end connection 127.0.0.5:47423 (5 connections now open) m31002| Fri Jan 18 13:12:21.437 [initandlisten] connection accepted from 127.0.0.5:40043 #165 (6 connections now open) m31001| Fri Jan 18 13:12:21.443 [initandlisten] connection accepted from 127.0.0.5:58004 #193 (6 connections now open) m31003| Fri Jan 18 13:12:21.591 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:21.591 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.591 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.591 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31002| Fri Jan 18 13:12:21.595 [initandlisten] connection accepted from 127.0.0.5:45031 #166 (7 connections now open) m31002| Fri Jan 18 13:12:21.596 [initandlisten] connection accepted from 127.0.0.5:59246 #167 (8 connections now open) m31002| Fri Jan 18 13:12:21.596 [conn165] end connection 127.0.0.5:40043 (7 connections now open) m31001| Fri Jan 18 13:12:21.624 [conn193] end connection 127.0.0.5:58004 (5 connections now open) m31003| Fri Jan 18 13:12:21.825 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:21.825 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:21.825 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:21.885 [conn166] end connection 127.0.0.5:45031 (6 connections now open) m31002| Fri Jan 18 13:12:21.975 [initandlisten] connection accepted from 127.0.0.5:55806 #168 (7 connections now open) m31002| Fri Jan 18 13:12:22.045 [conn167] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532593000|7400 } } cursorid:977394155649536 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:314950 nreturned:197502 reslen:3357554 160ms m31003| Fri Jan 18 13:12:22.098 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:22.098 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:22.098 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:22.178 [conn168] end connection 127.0.0.5:55806 (6 connections now open) m31002| Fri Jan 18 13:12:22.196 [initandlisten] connection accepted from 127.0.0.5:42394 #169 (7 connections now open) m31003| Fri Jan 18 13:12:22.264 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:22.264 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:22.264 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:22.276 [initandlisten] connection accepted from 127.0.0.5:48235 #170 (8 connections now open) m31002| Fri Jan 18 13:12:22.309 [conn169] end connection 127.0.0.5:42394 (7 connections now open) m31003| Fri Jan 18 13:12:22.333 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:22.333 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:22.333 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:22.367 [initandlisten] connection accepted from 127.0.0.5:56012 #171 (8 connections now open) m31002| Fri Jan 18 13:12:22.367 [conn170] end connection 127.0.0.5:48235 (7 connections now open) m31002| Fri Jan 18 13:12:22.453 [conn171] end connection 127.0.0.5:56012 (6 connections now open) m31003| Fri Jan 18 13:12:22.470 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:22.470 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:22.470 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:22.568 [initandlisten] connection accepted from 127.0.0.5:37164 #172 (7 connections now open) m31002| Fri Jan 18 13:12:22.754 [conn172] end connection 127.0.0.5:37164 (6 connections now open) m31003| Fri Jan 18 13:12:22.756 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:22.756 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:22.756 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:22.930 [initandlisten] connection accepted from 127.0.0.5:47543 #173 (7 connections now open) m31003| Fri Jan 18 13:12:23.018 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:23.018 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.018 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:23.086 [conn173] end connection 127.0.0.5:47543 (6 connections now open) m31002| Fri Jan 18 13:12:23.154 [initandlisten] connection accepted from 127.0.0.5:50338 #174 (7 connections now open) m31003| Fri Jan 18 13:12:23.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:23.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:23.279 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:23.279 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Fri Jan 18 13:12:23.295 [conn174] end connection 127.0.0.5:50338 (6 connections now open) m31001| Fri Jan 18 13:12:23.303 [initandlisten] connection accepted from 127.0.0.5:38400 #194 (6 connections now open) m31003| Fri Jan 18 13:12:23.399 [conn26] end connection 127.0.0.3:43485 (3 connections now open) m31003| Fri Jan 18 13:12:23.399 [initandlisten] connection accepted from 127.0.0.3:47918 #29 (4 connections now open) m31002| Fri Jan 18 13:12:23.447 [conn70] end connection 127.0.0.6:53787 (5 connections now open) m31002| Fri Jan 18 13:12:23.448 [initandlisten] connection accepted from 127.0.0.6:57914 #175 (6 connections now open) m31003| Fri Jan 18 13:12:23.468 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:23.468 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:23.468 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.468 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.468 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:23.470 [conn167] end connection 127.0.0.5:59246 (5 connections now open) m31001| Fri Jan 18 13:12:23.498 [conn194] end connection 127.0.0.5:38400 (5 connections now open) m31001| Fri Jan 18 13:12:23.552 [initandlisten] connection accepted from 127.0.0.5:52014 #195 (6 connections now open) m31002| Fri Jan 18 13:12:23.649 [initandlisten] connection accepted from 127.0.0.5:43665 #176 (6 connections now open) m31003| Fri Jan 18 13:12:23.711 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.711 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:23.711 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.711 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31001| Fri Jan 18 13:12:23.758 [conn195] end connection 127.0.0.5:52014 (5 connections now open) m31002| Fri Jan 18 13:12:23.807 [initandlisten] connection accepted from 127.0.0.5:34051 #177 (7 connections now open) m31002| Fri Jan 18 13:12:23.807 [initandlisten] connection accepted from 127.0.0.5:44685 #178 (8 connections now open) m31002| Fri Jan 18 13:12:23.807 [conn176] end connection 127.0.0.5:43665 (7 connections now open) m31003| Fri Jan 18 13:12:23.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Fri Jan 18 13:12:23.955 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.2, filling with zeroes... m31003| Fri Jan 18 13:12:23.967 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:23.967 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:23.967 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.029 [initandlisten] connection accepted from 127.0.0.5:58172 #179 (8 connections now open) m31002| Fri Jan 18 13:12:24.063 [conn177] end connection 127.0.0.5:34051 (7 connections now open) m31002| Fri Jan 18 13:12:24.145 [conn179] end connection 127.0.0.5:58172 (6 connections now open) m31003| Fri Jan 18 13:12:24.147 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:24.147 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:24.147 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.196 [conn178] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|291 } } cursorid:986714120605242 ntoreturn:0 keyUpdates:0 numYields: 3 locks(micros) r:274014 nreturned:208654 reslen:3547138 167ms m31002| Fri Jan 18 13:12:24.230 [initandlisten] connection accepted from 127.0.0.5:55210 #180 (7 connections now open) m31003| Fri Jan 18 13:12:24.324 [conn27] end connection 127.0.0.4:41882 (3 connections now open) m31003| Fri Jan 18 13:12:24.324 [initandlisten] connection accepted from 127.0.0.4:57469 #30 (4 connections now open) m31003| Fri Jan 18 13:12:24.372 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:24.372 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:24.372 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.384 [conn180] end connection 127.0.0.5:55210 (6 connections now open) m31002| Fri Jan 18 13:12:24.385 [initandlisten] connection accepted from 127.0.0.5:36411 #181 (7 connections now open) m31001| Fri Jan 18 13:12:24.444 [conn136] end connection 127.0.0.2:60375 (4 connections now open) m31001| Fri Jan 18 13:12:24.444 [initandlisten] connection accepted from 127.0.0.2:44076 #196 (5 connections now open) m31000| Fri Jan 18 13:12:24.489 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.2, size: 64MB, took 0.533 secs m31000| Fri Jan 18 13:12:24.489 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:534122 534ms m31004| Fri Jan 18 13:12:24.490 [FileAllocator] allocating new datafile /data/db/testReplSet-4/foo.2, filling with zeroes... m31003| Fri Jan 18 13:12:24.591 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:24.591 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:24.591 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.619 [initandlisten] connection accepted from 127.0.0.5:50187 #182 (8 connections now open) m31002| Fri Jan 18 13:12:24.648 [conn181] end connection 127.0.0.5:36411 (7 connections now open) m31002| Fri Jan 18 13:12:24.744 [conn182] end connection 127.0.0.5:50187 (6 connections now open) m31003| Fri Jan 18 13:12:24.746 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:24.746 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:24.746 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.828 [initandlisten] connection accepted from 127.0.0.5:46711 #183 (7 connections now open) m31002| Fri Jan 18 13:12:24.889 [conn183] end connection 127.0.0.5:46711 (6 connections now open) m31003| Fri Jan 18 13:12:24.891 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:24.891 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:24.891 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:24.955 [initandlisten] connection accepted from 127.0.0.5:53200 #184 (7 connections now open) m31004| Fri Jan 18 13:12:25.057 [FileAllocator] done allocating datafile /data/db/testReplSet-4/foo.2, size: 64MB, took 0.567 secs m31003| Fri Jan 18 13:12:25.119 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:25.119 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:25.119 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:25.193 [initandlisten] connection accepted from 127.0.0.5:44987 #185 (8 connections now open) m31002| Fri Jan 18 13:12:25.193 [conn184] end connection 127.0.0.5:53200 (6 connections now open) STATUS FROM : 127.0.0.2:31000 m31002| Fri Jan 18 13:12:25.292 [conn78] end connection 127.0.0.5:42860 (6 connections now open) m31002| Fri Jan 18 13:12:25.313 [initandlisten] connection accepted from 127.0.0.5:54722 #186 (7 connections now open) m31003| Fri Jan 18 13:12:25.343 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:25.343 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:25.343 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:25.362 [conn185] end connection 127.0.0.5:44987 (6 connections now open) m31000| Fri Jan 18 13:12:25.399 [conn48] end connection 127.0.0.3:36408 (11 connections now open) m31000| Fri Jan 18 13:12:25.399 [initandlisten] connection accepted from 127.0.0.3:42313 #51 (12 connections now open) m31002| Fri Jan 18 13:12:25.432 [initandlisten] connection accepted from 127.0.0.5:48731 #187 (7 connections now open) m31000| Fri Jan 18 13:12:25.447 [conn49] end connection 127.0.0.6:54782 (11 connections now open) m31000| Fri Jan 18 13:12:25.448 [initandlisten] connection accepted from 127.0.0.6:49331 #52 (12 connections now open) m31003| Fri Jan 18 13:12:25.542 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:25.542 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:25.542 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 235, "optime" : { "t" : 1358532745000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532743000, "i" : 5335 }, "optimeDate" : ISODate("2013-01-18T18:12:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 200, "optime" : { "t" : 1358532745000, "i" : 5894 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 235, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532594000, "i" : 1091 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 199, "optime" : { "t" : 1358532743000, "i" : 5339 }, "optimeDate" : ISODate("2013-01-18T18:12:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 199, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 199, "optime" : { "t" : 1358532594000, "i" : 491 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 73, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 199, "optime" : { "t" : 1358532743000, "i" : 5335 }, "optimeDate" : ISODate("2013-01-18T18:12:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:25Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 39 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 33 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532594000, "i" : 1191 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358532743000, "i" : 4634 }, "optimeDate" : ISODate("2013-01-18T18:12:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:25Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 200, "optime" : { "t" : 1358532745000, "i" : 5894 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532703000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:43Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 200, "optime" : { "t" : 1358532594000, "i" : 1091 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532745000, "i" : 7792 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:12:25.602 [conn187] end connection 127.0.0.5:48731 (6 connections now open) m31002| Fri Jan 18 13:12:25.683 [initandlisten] connection accepted from 127.0.0.5:60921 #188 (7 connections now open) m31003| Fri Jan 18 13:12:25.755 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:25.755 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:25.755 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:25.849 [conn188] end connection 127.0.0.5:60921 (6 connections now open) m31002| Fri Jan 18 13:12:25.849 [initandlisten] connection accepted from 127.0.0.5:40121 #189 (8 connections now open) m31003| Fri Jan 18 13:12:26.007 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:26.007 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:26.007 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:26.094 [initandlisten] connection accepted from 127.0.0.5:36431 #190 (8 connections now open) m31002| Fri Jan 18 13:12:26.094 [conn189] end connection 127.0.0.5:40121 (7 connections now open) m31000| Fri Jan 18 13:12:26.097 [conn50] end connection 127.0.0.4:58259 (11 connections now open) m31000| Fri Jan 18 13:12:26.097 [initandlisten] connection accepted from 127.0.0.4:40524 #53 (12 connections now open) m31003| Fri Jan 18 13:12:26.236 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:26.236 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:26.236 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:26.305 [initandlisten] connection accepted from 127.0.0.5:55375 #191 (8 connections now open) m31002| Fri Jan 18 13:12:26.305 [conn190] end connection 127.0.0.5:36431 (7 connections now open) m31003| Fri Jan 18 13:12:26.382 [rsMgr] replSet I don't see a primary and I can't elect myself m31003| Fri Jan 18 13:12:26.426 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:26.426 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:26.426 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:26.437 [conn191] end connection 127.0.0.5:55375 (6 connections now open) m31002| Fri Jan 18 13:12:26.474 [initandlisten] connection accepted from 127.0.0.5:54444 #192 (7 connections now open) m31003| Fri Jan 18 13:12:26.593 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:26.593 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:26.593 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:26.660 [conn192] end connection 127.0.0.5:54444 (6 connections now open) m31002| Fri Jan 18 13:12:26.744 [initandlisten] connection accepted from 127.0.0.5:57749 #193 (7 connections now open) m31003| Fri Jan 18 13:12:26.841 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:26.841 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:26.841 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:26.909 [initandlisten] connection accepted from 127.0.0.5:44269 #194 (8 connections now open) m31002| Fri Jan 18 13:12:26.928 [conn193] end connection 127.0.0.5:57749 (7 connections now open) m31003| Fri Jan 18 13:12:27.071 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:27.071 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:27.071 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:27.081 [conn194] end connection 127.0.0.5:44269 (6 connections now open) m31002| Fri Jan 18 13:12:27.106 [initandlisten] connection accepted from 127.0.0.5:34445 #195 (7 connections now open) m31003| Fri Jan 18 13:12:27.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:27.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:27.230 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:27.299 [conn195] end connection 127.0.0.5:34445 (6 connections now open) m31002| Fri Jan 18 13:12:27.354 [initandlisten] connection accepted from 127.0.0.5:53649 #196 (7 connections now open) m31002| Fri Jan 18 13:12:27.498 [conn196] end connection 127.0.0.5:53649 (6 connections now open) m31003| Fri Jan 18 13:12:27.500 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:27.500 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:27.500 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:27.500 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:27.500 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:12:27.571 [initandlisten] connection accepted from 127.0.0.5:59642 #197 (6 connections now open) m31003| Fri Jan 18 13:12:27.641 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:27.641 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:27.641 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:27.641 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:27.641 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:27.733 [conn197] end connection 127.0.0.5:59642 (5 connections now open) m31001| Fri Jan 18 13:12:27.739 [initandlisten] connection accepted from 127.0.0.5:49618 #198 (6 connections now open) m31002| Fri Jan 18 13:12:27.741 [conn178] end connection 127.0.0.5:44685 (5 connections now open) m31001| Fri Jan 18 13:12:27.756 [initandlisten] connection accepted from 127.0.0.5:52529 #199 (7 connections now open) m31003| Fri Jan 18 13:12:27.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:27.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:27.839 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:27.839 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:27.893 [conn198] end connection 127.0.0.5:49618 (6 connections now open) m31001| Fri Jan 18 13:12:27.903 [initandlisten] connection accepted from 127.0.0.5:56918 #200 (7 connections now open) m31001| Fri Jan 18 13:12:28.169 [conn200] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|2291 } } cursorid:1007380619517381 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 820 locks(micros) r:125326 nreturned:101 reslen:10423 252ms m31003| Fri Jan 18 13:12:28.172 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:28.172 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:28.172 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:28.172 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:28.225 [conn199] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|2191 } } cursorid:1005938000049696 ntoreturn:0 keyUpdates:0 numYields: 1063 locks(micros) r:359114 nreturned:214132 reslen:3640264 331ms m31001| Fri Jan 18 13:12:28.258 [conn200] end connection 127.0.0.5:56918 (6 connections now open) m31001| Fri Jan 18 13:12:28.330 [initandlisten] connection accepted from 127.0.0.5:44064 #201 (7 connections now open) m31003| Fri Jan 18 13:12:28.535 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:28.535 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:28.535 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:28.535 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:28.571 [conn201] end connection 127.0.0.5:44064 (6 connections now open) m31001| Fri Jan 18 13:12:28.582 [initandlisten] connection accepted from 127.0.0.5:46706 #202 (7 connections now open) m31003| Fri Jan 18 13:12:28.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:28.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:28.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:28.720 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:28.736 [initandlisten] connection accepted from 127.0.0.5:44293 #203 (8 connections now open) m31001| Fri Jan 18 13:12:28.780 [conn202] end connection 127.0.0.5:46706 (7 connections now open) m31003| Fri Jan 18 13:12:28.838 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:28.838 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:28.838 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:28.838 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:28.872 [initandlisten] connection accepted from 127.0.0.5:58238 #204 (8 connections now open) m31001| Fri Jan 18 13:12:28.872 [conn203] end connection 127.0.0.5:44293 (7 connections now open) m31003| Fri Jan 18 13:12:29.021 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:29.021 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.021 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:29.021 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:29.086 [initandlisten] connection accepted from 127.0.0.5:46363 #205 (8 connections now open) m31001| Fri Jan 18 13:12:29.086 [conn204] end connection 127.0.0.5:58238 (6 connections now open) m31003| Fri Jan 18 13:12:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.241 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:29.241 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:29.296 [conn205] end connection 127.0.0.5:46363 (6 connections now open) m31001| Fri Jan 18 13:12:29.372 [initandlisten] connection accepted from 127.0.0.5:51441 #206 (7 connections now open) m31001| Fri Jan 18 13:12:29.490 [conn206] end connection 127.0.0.5:51441 (6 connections now open) m31003| Fri Jan 18 13:12:29.507 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:29.507 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.507 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.507 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:29.600 [initandlisten] connection accepted from 127.0.0.5:59592 #197 (6 connections now open) m31002| Fri Jan 18 13:12:29.669 [conn197] end connection 127.0.0.5:59592 (5 connections now open) m31003| Fri Jan 18 13:12:29.671 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:29.671 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.671 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.671 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Fri Jan 18 13:12:29.722 [conn199] end connection 127.0.0.5:52529 (5 connections now open) m31002| Fri Jan 18 13:12:29.723 [initandlisten] connection accepted from 127.0.0.5:49682 #198 (6 connections now open) m31002| Fri Jan 18 13:12:29.723 [initandlisten] connection accepted from 127.0.0.5:46097 #199 (7 connections now open) m31003| Fri Jan 18 13:12:29.856 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:29.856 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:29.856 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:29.936 [initandlisten] connection accepted from 127.0.0.5:39842 #200 (8 connections now open) m31002| Fri Jan 18 13:12:29.936 [conn198] end connection 127.0.0.5:49682 (7 connections now open) m31002| Fri Jan 18 13:12:30.127 [conn199] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|3091 } } cursorid:1011376583709757 ntoreturn:0 keyUpdates:0 numYields: 2 locks(micros) r:373514 nreturned:228235 reslen:3880015 190ms m31003| Fri Jan 18 13:12:30.173 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:30.173 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:30.173 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:30.269 [initandlisten] connection accepted from 127.0.0.5:52558 #201 (8 connections now open) m31002| Fri Jan 18 13:12:30.269 [conn200] end connection 127.0.0.5:39842 (7 connections now open) m31003| Fri Jan 18 13:12:30.339 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:30.339 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:30.339 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:30.361 [initandlisten] connection accepted from 127.0.0.5:33977 #202 (8 connections now open) m31002| Fri Jan 18 13:12:30.394 [conn201] end connection 127.0.0.5:52558 (7 connections now open) m31002| Fri Jan 18 13:12:30.531 [conn202] end connection 127.0.0.5:33977 (6 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Fri Jan 18 13:12:31.041 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:1608836 804ms m31003| Fri Jan 18 13:12:31.181 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:31.181 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:31.181 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:31.280 [initandlisten] connection accepted from 127.0.0.5:46507 #203 (7 connections now open) m31002| Fri Jan 18 13:12:31.457 [conn203] end connection 127.0.0.5:46507 (6 connections now open) m31003| Fri Jan 18 13:12:31.560 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:31.560 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:31.560 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:31.560 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:31.560 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:12:31.565 [initandlisten] connection accepted from 127.0.0.5:54110 #207 (6 connections now open) m31000| Fri Jan 18 13:12:31.574 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:495368 247ms { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 241, "optime" : { "t" : 1358532751000, "i" : 5315 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532750000, "i" : 3801 }, "optimeDate" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 206, "optime" : { "t" : 1358532751000, "i" : 3743 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 241, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532594000, "i" : 3491 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532751000, "i" : 3743 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 205, "optime" : { "t" : 1358532750000, "i" : 1450 }, "optimeDate" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 205, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 240, "optime" : { "t" : 1358532711000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:51Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 205, "optime" : { "t" : 1358532594000, "i" : 3291 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 36, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 205, "optime" : { "t" : 1358532750000, "i" : 1423 }, "optimeDate" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:31Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 24 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 24 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532594000, "i" : 3591 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358532751000, "i" : 3743 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:31Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:31.656 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:31.656 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:31.656 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:31.656 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:31.656 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:31Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 206, "optime" : { "t" : 1358532751000, "i" : 3743 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532708000, "i" : 5188 }, "optimeDate" : ISODate("2013-01-18T18:11:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358532594000, "i" : 3491 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532751000, "i" : 5315 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "self" : true } ], "ok" : 1 } m31001| Fri Jan 18 13:12:31.662 [conn207] end connection 127.0.0.5:54110 (5 connections now open) m31001| Fri Jan 18 13:12:31.711 [initandlisten] connection accepted from 127.0.0.5:59494 #208 (6 connections now open) m31002| Fri Jan 18 13:12:31.725 [conn199] end connection 127.0.0.5:46097 (5 connections now open) { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:12:31.796 [initandlisten] connection accepted from 127.0.0.5:51835 #209 (7 connections now open) m31003| Fri Jan 18 13:12:31.887 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:31.887 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:31.887 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:31.887 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:31.913 [conn209] end connection 127.0.0.5:51835 (6 connections now open) m31001| Fri Jan 18 13:12:31.913 [initandlisten] connection accepted from 127.0.0.5:55903 #210 (8 connections now open) m31003| Fri Jan 18 13:12:32.002 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.002 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.002 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.002 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:32.016 [conn210] end connection 127.0.0.5:55903 (6 connections now open) m31001| Fri Jan 18 13:12:32.016 [initandlisten] connection accepted from 127.0.0.5:44237 #211 (8 connections now open) m31003| Fri Jan 18 13:12:32.080 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.080 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.080 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.080 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:32.100 [conn208] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|3691 } } cursorid:1023129971794576 ntoreturn:0 keyUpdates:0 numYields: 8 locks(micros) r:344786 nreturned:228101 reslen:3877737 187ms m31001| Fri Jan 18 13:12:32.159 [conn211] end connection 127.0.0.5:44237 (6 connections now open) m31001| Fri Jan 18 13:12:32.186 [initandlisten] connection accepted from 127.0.0.5:60148 #212 (7 connections now open) m31003| Fri Jan 18 13:12:32.314 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.314 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.314 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.314 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:32.379 [conn212] end connection 127.0.0.5:60148 (6 connections now open) m31001| Fri Jan 18 13:12:32.394 [initandlisten] connection accepted from 127.0.0.5:49337 #213 (7 connections now open) m31000| Fri Jan 18 13:12:32.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Fri Jan 18 13:12:32.559 [conn213] end connection 127.0.0.5:49337 (6 connections now open) m31003| Fri Jan 18 13:12:32.561 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.561 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.561 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.561 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:32.655 [initandlisten] connection accepted from 127.0.0.5:34409 #214 (7 connections now open) m31003| Fri Jan 18 13:12:32.806 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.806 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.806 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.806 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:32.898 [conn214] end connection 127.0.0.5:34409 (6 connections now open) m31001| Fri Jan 18 13:12:32.898 [initandlisten] connection accepted from 127.0.0.5:53144 #215 (8 connections now open) m31003| Fri Jan 18 13:12:32.982 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:32.982 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:32.982 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:32.982 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:33.073 [conn215] end connection 127.0.0.5:53144 (6 connections now open) m31001| Fri Jan 18 13:12:33.073 [initandlisten] connection accepted from 127.0.0.5:55802 #216 (8 connections now open) m31001| Fri Jan 18 13:12:33.170 [conn216] end connection 127.0.0.5:55802 (6 connections now open) m31003| Fri Jan 18 13:12:33.614 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:33.614 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:33.614 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:33.614 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:33.698 [initandlisten] connection accepted from 127.0.0.5:43122 #217 (7 connections now open) m31003| Fri Jan 18 13:12:33.937 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:33.937 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:33.937 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:33.937 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:33.994 [conn217] end connection 127.0.0.5:43122 (6 connections now open) m31001| Fri Jan 18 13:12:33.996 [initandlisten] connection accepted from 127.0.0.5:34824 #218 (7 connections now open) m31001| Fri Jan 18 13:12:34.269 [conn218] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|4591 } } cursorid:1033313750015355 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 932 locks(micros) r:138281 nreturned:101 reslen:10423 186ms m31003| Fri Jan 18 13:12:34.273 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:34.273 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:34.273 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:34.273 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:34.368 [initandlisten] connection accepted from 127.0.0.5:51566 #219 (8 connections now open) m31001| Fri Jan 18 13:12:34.368 [conn218] end connection 127.0.0.5:34824 (7 connections now open) m31003| Fri Jan 18 13:12:34.494 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:34.494 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:34.494 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:34.494 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:34.505 [conn219] end connection 127.0.0.5:51566 (6 connections now open) m31001| Fri Jan 18 13:12:34.642 [initandlisten] connection accepted from 127.0.0.5:47529 #220 (7 connections now open) m31003| Fri Jan 18 13:12:34.727 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:34.727 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:34.727 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:34.727 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:34.787 [conn220] end connection 127.0.0.5:47529 (6 connections now open) m31001| Fri Jan 18 13:12:34.828 [initandlisten] connection accepted from 127.0.0.5:36405 #221 (7 connections now open) m31003| Fri Jan 18 13:12:34.914 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:34.914 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:34.914 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:34.914 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:34.937 [conn221] end connection 127.0.0.5:36405 (6 connections now open) m31001| Fri Jan 18 13:12:34.956 [initandlisten] connection accepted from 127.0.0.5:56151 #222 (7 connections now open) m31003| Fri Jan 18 13:12:35.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:35.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:35.024 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:35.024 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:35.118 [conn222] end connection 127.0.0.5:56151 (6 connections now open) m31001| Fri Jan 18 13:12:35.118 [initandlisten] connection accepted from 127.0.0.5:40981 #223 (8 connections now open) m31003| Fri Jan 18 13:12:35.204 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:35.204 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:35.204 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:35.204 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:35.285 [initandlisten] connection accepted from 127.0.0.5:58158 #224 (8 connections now open) m31001| Fri Jan 18 13:12:35.285 [conn223] end connection 127.0.0.5:40981 (7 connections now open) m31003| Fri Jan 18 13:12:35.449 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:35.449 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:35.449 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:35.449 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:35.477 [initandlisten] connection accepted from 127.0.0.5:45120 #204 (6 connections now open) m31001| Fri Jan 18 13:12:35.512 [conn224] end connection 127.0.0.5:58158 (6 connections now open) m31003| Fri Jan 18 13:12:35.569 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31002| Fri Jan 18 13:12:35.627 [conn204] end connection 127.0.0.5:45120 (5 connections now open) m31001| Fri Jan 18 13:12:35.662 [conn208] end connection 127.0.0.5:59494 (5 connections now open) m31002| Fri Jan 18 13:12:35.705 [initandlisten] connection accepted from 127.0.0.5:50372 #205 (6 connections now open) m31003| Fri Jan 18 13:12:35.757 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:35.757 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:35.757 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:35.930 [initandlisten] connection accepted from 127.0.0.5:45796 #206 (7 connections now open) m31003| Fri Jan 18 13:12:36.027 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:36.027 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:36.027 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:36.054 [conn206] end connection 127.0.0.5:45796 (6 connections now open) m31002| Fri Jan 18 13:12:36.153 [initandlisten] connection accepted from 127.0.0.5:34575 #207 (7 connections now open) m31002| Fri Jan 18 13:12:36.253 [conn205] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|5391 } } cursorid:1037330201769727 ntoreturn:0 keyUpdates:0 locks(micros) r:198624 nreturned:242398 reslen:4120786 198ms m31003| Fri Jan 18 13:12:36.302 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:36.302 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:36.302 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:36.330 [conn207] end connection 127.0.0.5:34575 (6 connections now open) m31002| Fri Jan 18 13:12:36.347 [initandlisten] connection accepted from 127.0.0.5:46265 #208 (7 connections now open) m31004| Fri Jan 18 13:12:36.446 [conn30] end connection 127.0.0.2:49811 (4 connections now open) m31004| Fri Jan 18 13:12:36.446 [initandlisten] connection accepted from 127.0.0.2:58263 #33 (5 connections now open) m31003| Fri Jan 18 13:12:36.498 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:36.498 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:36.498 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:36.530 [initandlisten] connection accepted from 127.0.0.5:39867 #209 (8 connections now open) m31002| Fri Jan 18 13:12:36.538 [conn208] end connection 127.0.0.5:46265 (7 connections now open) m31003| Fri Jan 18 13:12:36.615 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:36.615 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:36.615 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:36.632 [conn209] end connection 127.0.0.5:39867 (6 connections now open) m31002| Fri Jan 18 13:12:36.689 [initandlisten] connection accepted from 127.0.0.5:35521 #210 (7 connections now open) m31003| Fri Jan 18 13:12:36.849 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:36.849 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:36.849 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:36.906 [conn210] end connection 127.0.0.5:35521 (6 connections now open) m31002| Fri Jan 18 13:12:36.946 [initandlisten] connection accepted from 127.0.0.5:39452 #211 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 247, "optime" : { "t" : 1358532757000, "i" : 612 }, "optimeDate" : ISODate("2013-01-18T18:12:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532756000, "i" : 4239 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 212, "optime" : { "t" : 1358532755000, "i" : 3676 }, "optimeDate" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 247, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532594000, "i" : 5191 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532755000, "i" : 3628 }, "optimeDate" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:35Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 211, "optime" : { "t" : 1358532756000, "i" : 1109 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 211, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 246, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 211, "optime" : { "t" : 1358532594000, "i" : 5591 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 17, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 211, "optime" : { "t" : 1358532756000, "i" : 1098 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:37Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 25 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 15 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 245, "optime" : { "t" : 1358532594000, "i" : 5891 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 213, "optime" : { "t" : 1358532756000, "i" : 3582 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.066 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.066 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:37.066 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 212, "optime" : { "t" : 1358532755000, "i" : 4247 }, "optimeDate" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532713000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:11:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358532594000, "i" : 5268 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:36Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 245, "optime" : { "t" : 1358532757000, "i" : 612 }, "optimeDate" : ISODate("2013-01-18T18:12:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:12:37.105 [conn211] end connection 127.0.0.5:39452 (6 connections now open) m31002| Fri Jan 18 13:12:37.124 [initandlisten] connection accepted from 127.0.0.5:44321 #212 (7 connections now open) m31003| Fri Jan 18 13:12:37.399 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.399 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:37.399 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31004| Fri Jan 18 13:12:37.401 [conn31] end connection 127.0.0.3:39519 (4 connections now open) m31004| Fri Jan 18 13:12:37.401 [initandlisten] connection accepted from 127.0.0.3:49490 #34 (5 connections now open) m31002| Fri Jan 18 13:12:37.439 [conn212] end connection 127.0.0.5:44321 (6 connections now open) m31003| Fri Jan 18 13:12:37.449 [conn28] end connection 127.0.0.6:34386 (3 connections now open) m31003| Fri Jan 18 13:12:37.450 [initandlisten] connection accepted from 127.0.0.6:56478 #31 (4 connections now open) m31002| Fri Jan 18 13:12:37.481 [initandlisten] connection accepted from 127.0.0.5:54034 #213 (7 connections now open) m31003| Fri Jan 18 13:12:37.555 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.555 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:37.555 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:37.634 [initandlisten] connection accepted from 127.0.0.5:38347 #214 (8 connections now open) m31002| Fri Jan 18 13:12:37.634 [conn213] end connection 127.0.0.5:54034 (6 connections now open) m31003| Fri Jan 18 13:12:37.785 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.785 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:37.785 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:37.835 [conn214] end connection 127.0.0.5:38347 (6 connections now open) m31002| Fri Jan 18 13:12:37.886 [initandlisten] connection accepted from 127.0.0.5:41629 #215 (7 connections now open) m31003| Fri Jan 18 13:12:37.980 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:37.980 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:37.980 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:38.056 [conn215] end connection 127.0.0.5:41629 (6 connections now open) m31002| Fri Jan 18 13:12:38.056 [initandlisten] connection accepted from 127.0.0.5:33220 #216 (8 connections now open) m31004| Fri Jan 18 13:12:38.099 [conn32] end connection 127.0.0.4:48460 (4 connections now open) m31004| Fri Jan 18 13:12:38.099 [initandlisten] connection accepted from 127.0.0.4:39764 #35 (5 connections now open) m31003| Fri Jan 18 13:12:38.269 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:38.269 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:38.269 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:38.349 [conn216] end connection 127.0.0.5:33220 (6 connections now open) m31002| Fri Jan 18 13:12:38.361 [initandlisten] connection accepted from 127.0.0.5:51690 #217 (7 connections now open) m31002| Fri Jan 18 13:12:38.446 [conn123] end connection 127.0.0.2:37295 (6 connections now open) m31002| Fri Jan 18 13:12:38.446 [initandlisten] connection accepted from 127.0.0.2:53080 #218 (8 connections now open) m31003| Fri Jan 18 13:12:38.564 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:38.564 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:38.564 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:38.632 [conn217] end connection 127.0.0.5:51690 (6 connections now open) m31002| Fri Jan 18 13:12:38.667 [initandlisten] connection accepted from 127.0.0.5:48319 #219 (7 connections now open) m31003| Fri Jan 18 13:12:38.837 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:38.837 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:38.837 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:38.877 [conn219] end connection 127.0.0.5:48319 (6 connections now open) m31002| Fri Jan 18 13:12:38.882 [initandlisten] connection accepted from 127.0.0.5:54384 #220 (7 connections now open) m31003| Fri Jan 18 13:12:39.150 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:39.150 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:39.150 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:39.166 [initandlisten] connection accepted from 127.0.0.5:58393 #221 (8 connections now open) m31002| Fri Jan 18 13:12:39.166 [conn220] end connection 127.0.0.5:54384 (6 connections now open) m31002| Fri Jan 18 13:12:39.399 [conn221] end connection 127.0.0.5:58393 (6 connections now open) m31002| Fri Jan 18 13:12:39.401 [conn124] end connection 127.0.0.3:58882 (5 connections now open) m31002| Fri Jan 18 13:12:39.401 [initandlisten] connection accepted from 127.0.0.3:47540 #222 (6 connections now open) m31003| Fri Jan 18 13:12:39.403 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:39.403 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:39.403 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:39.437 [initandlisten] connection accepted from 127.0.0.5:40983 #223 (7 connections now open) m31001| Fri Jan 18 13:12:39.450 [conn171] end connection 127.0.0.6:52171 (4 connections now open) m31001| Fri Jan 18 13:12:39.450 [initandlisten] connection accepted from 127.0.0.6:40655 #225 (5 connections now open) m31002| Fri Jan 18 13:12:39.525 [conn223] end connection 127.0.0.5:40983 (6 connections now open) m31003| Fri Jan 18 13:12:39.529 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:39.529 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:39.529 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:39.529 [initandlisten] connection accepted from 127.0.0.5:44464 #224 (7 connections now open) m31003| Fri Jan 18 13:12:39.664 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:39.664 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:39.664 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:39.734 [initandlisten] connection accepted from 127.0.0.5:35901 #225 (8 connections now open) m31002| Fri Jan 18 13:12:39.734 [conn224] end connection 127.0.0.5:44464 (6 connections now open) m31003| Fri Jan 18 13:12:39.929 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:39.929 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:39.929 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:39.989 [initandlisten] connection accepted from 127.0.0.5:58490 #226 (8 connections now open) m31002| Fri Jan 18 13:12:39.989 [conn225] end connection 127.0.0.5:35901 (7 connections now open) m31001| Fri Jan 18 13:12:40.099 [conn176] end connection 127.0.0.4:52897 (4 connections now open) m31001| Fri Jan 18 13:12:40.099 [initandlisten] connection accepted from 127.0.0.4:38759 #226 (5 connections now open) m31002| Fri Jan 18 13:12:40.175 [conn226] end connection 127.0.0.5:58490 (6 connections now open) m31003| Fri Jan 18 13:12:40.177 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:40.177 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:40.177 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:40.205 [initandlisten] connection accepted from 127.0.0.5:36664 #227 (7 connections now open) m31002| Fri Jan 18 13:12:40.340 [conn227] end connection 127.0.0.5:36664 (6 connections now open) m31003| Fri Jan 18 13:12:40.345 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:40.345 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:40.345 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:40.449 [initandlisten] connection accepted from 127.0.0.5:33629 #228 (7 connections now open) m31003| Fri Jan 18 13:12:40.624 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:40.624 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:40.624 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:40.686 [conn228] end connection 127.0.0.5:33629 (6 connections now open) m31002| Fri Jan 18 13:12:40.707 [initandlisten] connection accepted from 127.0.0.5:41934 #229 (7 connections now open) m31003| Fri Jan 18 13:12:40.787 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:40.787 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:40.787 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:40.861 [conn229] end connection 127.0.0.5:41934 (6 connections now open) m31002| Fri Jan 18 13:12:40.933 [initandlisten] connection accepted from 127.0.0.5:44668 #230 (7 connections now open) m31003| Fri Jan 18 13:12:41.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:41.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:41.009 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:41.072 [initandlisten] connection accepted from 127.0.0.5:33830 #231 (8 connections now open) m31002| Fri Jan 18 13:12:41.072 [conn230] end connection 127.0.0.5:44668 (6 connections now open) m31002| Fri Jan 18 13:12:41.228 [conn231] end connection 127.0.0.5:33830 (6 connections now open) m31003| Fri Jan 18 13:12:41.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:41.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:41.230 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:41.312 [initandlisten] connection accepted from 127.0.0.5:36198 #232 (7 connections now open) m31003| Fri Jan 18 13:12:41.418 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:41.418 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:41.418 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:41.512 [initandlisten] connection accepted from 127.0.0.5:54057 #233 (8 connections now open) m31002| Fri Jan 18 13:12:41.512 [conn232] end connection 127.0.0.5:36198 (6 connections now open) m31001| Fri Jan 18 13:12:41.539 [conn180] end connection 127.0.0.5:44651 (4 connections now open) m31001| Fri Jan 18 13:12:41.645 [initandlisten] connection accepted from 127.0.0.5:35520 #227 (5 connections now open) m31003| Fri Jan 18 13:12:41.703 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:41.703 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:41.703 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:41.703 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:41.703 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Fri Jan 18 13:12:41.740 [conn233] end connection 127.0.0.5:54057 (6 connections now open) m31001| Fri Jan 18 13:12:41.766 [initandlisten] connection accepted from 127.0.0.5:34222 #228 (6 connections now open) m31003| Fri Jan 18 13:12:41.928 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:41.928 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:41.928 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:41.928 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:41.928 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:12:41.928 [conn205] end connection 127.0.0.5:50372 (5 connections now open) m31001| Fri Jan 18 13:12:41.986 [conn228] end connection 127.0.0.5:34222 (5 connections now open) m31001| Fri Jan 18 13:12:42.065 [initandlisten] connection accepted from 127.0.0.5:53378 #229 (6 connections now open) m31001| Fri Jan 18 13:12:42.065 [initandlisten] connection accepted from 127.0.0.5:53420 #230 (7 connections now open) m31003| Fri Jan 18 13:12:42.217 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:42.217 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:42.217 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:42.217 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:42.242 [conn230] end connection 127.0.0.5:53420 (6 connections now open) m31001| Fri Jan 18 13:12:42.260 [initandlisten] connection accepted from 127.0.0.5:46443 #231 (7 connections now open) STATUS FROM : 127.0.0.2:31000 m31000| Fri Jan 18 13:12:42.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Fri Jan 18 13:12:42.444 [conn229] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532594000|8091 } } cursorid:1067172669737571 ntoreturn:0 keyUpdates:0 numYields: 5 locks(micros) r:397953 nreturned:246722 reslen:4194294 201ms { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 252, "optime" : { "t" : 1358532762000, "i" : 3513 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532762000, "i" : 3440 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 217, "optime" : { "t" : 1358532761000, "i" : 3102 }, "optimeDate" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 252, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532594000, "i" : 7791 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:41Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532761000, "i" : 3089 }, "optimeDate" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:41Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 216, "optime" : { "t" : 1358532762000, "i" : 727 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 251, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358532594000, "i" : 8191 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 8, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358532762000, "i" : 711 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:42Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 28 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 31 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 250, "optime" : { "t" : 1358532594000, "i" : 8191 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 218, "optime" : { "t" : 1358532761000, "i" : 2741 }, "optimeDate" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:41Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:42.469 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:42.469 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:42.469 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:42.469 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 217, "optime" : { "t" : 1358532761000, "i" : 3609 }, "optimeDate" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532719000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:11:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:42Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358532594000, "i" : 7891 }, "optimeDate" : ISODate("2013-01-18T18:09:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 250, "optime" : { "t" : 1358532762000, "i" : 3513 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:12:42.528 [initandlisten] connection accepted from 127.0.0.5:44808 #232 (8 connections now open) m31001| Fri Jan 18 13:12:42.528 [conn231] end connection 127.0.0.5:46443 (6 connections now open) m31003| Fri Jan 18 13:12:42.700 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:42.700 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:42.700 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:42.700 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:42.702 [conn232] end connection 127.0.0.5:44808 (6 connections now open) m31001| Fri Jan 18 13:12:42.777 [initandlisten] connection accepted from 127.0.0.5:51906 #233 (7 connections now open) m31003| Fri Jan 18 13:12:42.872 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:42.872 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:42.872 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:42.872 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:42.940 [conn233] end connection 127.0.0.5:51906 (6 connections now open) m31001| Fri Jan 18 13:12:42.941 [initandlisten] connection accepted from 127.0.0.5:42537 #234 (7 connections now open) m31001| Fri Jan 18 13:12:43.366 [conn234] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532595000|165 } } cursorid:1072050160453222 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1050 locks(micros) r:178734 nreturned:101 reslen:10423 216ms m31001| Fri Jan 18 13:12:43.367 [conn234] end connection 127.0.0.5:42537 (6 connections now open) m31003| Fri Jan 18 13:12:43.370 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:43.370 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:43.370 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:43.370 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:43.523 [initandlisten] connection accepted from 127.0.0.5:54983 #235 (7 connections now open) m31003| Fri Jan 18 13:12:43.696 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:43.696 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:43.696 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:12:43.696 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:12:43.748 [initandlisten] connection accepted from 127.0.0.5:53401 #234 (6 connections now open) m31001| Fri Jan 18 13:12:43.787 [conn235] end connection 127.0.0.5:54983 (6 connections now open) m31004| Fri Jan 18 13:12:43.884 [conn29] end connection 127.0.0.5:41527 (4 connections now open) m31003| Fri Jan 18 13:12:43.898 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31002| Fri Jan 18 13:12:43.970 [conn234] end connection 127.0.0.5:53401 (5 connections now open) m31002| Fri Jan 18 13:12:43.970 [initandlisten] connection accepted from 127.0.0.5:44611 #235 (7 connections now open) m31001| Fri Jan 18 13:12:43.992 [conn229] end connection 127.0.0.5:53378 (5 connections now open) m31003| Fri Jan 18 13:12:43.999 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:43.999 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:43.999 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:44.031 [initandlisten] connection accepted from 127.0.0.5:35793 #236 (7 connections now open) m31002| Fri Jan 18 13:12:44.247 [conn236] end connection 127.0.0.5:35793 (6 connections now open) m31003| Fri Jan 18 13:12:44.250 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:44.250 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:44.250 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31004| Fri Jan 18 13:12:44.384 [initandlisten] connection accepted from 127.0.0.5:37741 #36 (5 connections now open) m31002| Fri Jan 18 13:12:44.446 [initandlisten] connection accepted from 127.0.0.5:47534 #237 (7 connections now open) m31002| Fri Jan 18 13:12:44.538 [conn235] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532595000|465 } } cursorid:1072308675760240 ntoreturn:0 keyUpdates:0 numYields: 91 locks(micros) r:354713 nreturned:246722 reslen:4194294 220ms m31003| Fri Jan 18 13:12:44.547 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:44.547 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:44.547 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:44.633 [conn237] end connection 127.0.0.5:47534 (6 connections now open) m31002| Fri Jan 18 13:12:44.633 [initandlisten] connection accepted from 127.0.0.5:52322 #238 (8 connections now open) m31003| Fri Jan 18 13:12:44.793 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:44.793 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:44.793 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:44.839 [initandlisten] connection accepted from 127.0.0.5:36178 #239 (8 connections now open) m31002| Fri Jan 18 13:12:44.839 [conn238] end connection 127.0.0.5:52322 (7 connections now open) m31003| Fri Jan 18 13:12:44.958 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:44.958 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:44.958 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:45.015 [conn239] end connection 127.0.0.5:36178 (6 connections now open) m31002| Fri Jan 18 13:12:45.015 [initandlisten] connection accepted from 127.0.0.5:57979 #240 (8 connections now open) m31002| Fri Jan 18 13:12:45.227 [conn240] end connection 127.0.0.5:57979 (6 connections now open) m31003| Fri Jan 18 13:12:45.229 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:45.229 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:45.229 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:45.276 [initandlisten] connection accepted from 127.0.0.5:48547 #241 (7 connections now open) m31002| Fri Jan 18 13:12:45.492 [conn241] end connection 127.0.0.5:48547 (6 connections now open) m31003| Fri Jan 18 13:12:45.661 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:45.661 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:45.661 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31000| Fri Jan 18 13:12:45.677 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:313403 156ms m31002| Fri Jan 18 13:12:45.721 [initandlisten] connection accepted from 127.0.0.5:57338 #242 (7 connections now open) m31003| Fri Jan 18 13:12:45.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Fri Jan 18 13:12:45.913 [conn242] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532595000|1065 } } cursorid:1079967171624305 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1087 locks(micros) r:205459 nreturned:101 reslen:10423 104ms m31003| Fri Jan 18 13:12:45.916 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:45.916 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:45.916 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:45.983 [initandlisten] connection accepted from 127.0.0.5:36162 #243 (8 connections now open) m31002| Fri Jan 18 13:12:45.983 [conn242] end connection 127.0.0.5:57338 (6 connections now open) m31003| Fri Jan 18 13:12:46.134 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:46.134 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:46.134 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:46.182 [conn243] end connection 127.0.0.5:36162 (6 connections now open) m31002| Fri Jan 18 13:12:46.232 [initandlisten] connection accepted from 127.0.0.5:42362 #244 (7 connections now open) m31003| Fri Jan 18 13:12:46.469 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:46.469 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:46.469 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:46.517 [conn244] end connection 127.0.0.5:42362 (6 connections now open) m31002| Fri Jan 18 13:12:46.553 [initandlisten] connection accepted from 127.0.0.5:36922 #245 (7 connections now open) m31002| Fri Jan 18 13:12:46.676 [conn245] end connection 127.0.0.5:36922 (6 connections now open) m31003| Fri Jan 18 13:12:46.678 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:46.678 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:46.678 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:46.716 [initandlisten] connection accepted from 127.0.0.5:48162 #246 (7 connections now open) m31002| Fri Jan 18 13:12:46.940 [conn246] end connection 127.0.0.5:48162 (6 connections now open) m31003| Fri Jan 18 13:12:46.943 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:46.943 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:46.943 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:47.048 [initandlisten] connection accepted from 127.0.0.5:44283 #247 (7 connections now open) m31003| Fri Jan 18 13:12:47.203 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:47.203 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:47.203 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:47.258 [conn247] end connection 127.0.0.5:44283 (6 connections now open) m31002| Fri Jan 18 13:12:47.270 [initandlisten] connection accepted from 127.0.0.5:38375 #248 (7 connections now open) m31003| Fri Jan 18 13:12:47.404 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:47.404 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:47.404 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:47.451 [conn248] end connection 127.0.0.5:38375 (6 connections now open) m31002| Fri Jan 18 13:12:47.458 [initandlisten] connection accepted from 127.0.0.5:60202 #249 (7 connections now open) STATUS FROM : 127.0.0.2:31000 m31003| Fri Jan 18 13:12:47.578 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:47.578 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:47.578 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:47.609 [initandlisten] connection accepted from 127.0.0.5:53325 #250 (8 connections now open) m31002| Fri Jan 18 13:12:47.609 [conn249] end connection 127.0.0.5:60202 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:47Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 257, "optime" : { "t" : 1358532767000, "i" : 5820 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532766000, "i" : 4347 }, "optimeDate" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 222, "optime" : { "t" : 1358532767000, "i" : 4403 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 257, "optime" : { "t" : 1358532724000, "i" : 2 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532595000, "i" : 1665 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532767000, "i" : 4398 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 221, "optime" : { "t" : 1358532766000, "i" : 1291 }, "optimeDate" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:46Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 256, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358532595000, "i" : 1265 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 4, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358532766000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:47Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 34 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 23 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532595000, "i" : 1865 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 223, "optime" : { "t" : 1358532766000, "i" : 7320 }, "optimeDate" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:47Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:47Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 222, "optime" : { "t" : 1358532767000, "i" : 4669 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532724000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:04Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358532595000, "i" : 1765 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532767000, "i" : 5820 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:12:47.806 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:47.806 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:47.806 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:47.867 [conn250] end connection 127.0.0.5:53325 (6 connections now open) m31002| Fri Jan 18 13:12:47.913 [initandlisten] connection accepted from 127.0.0.5:45295 #251 (7 connections now open) m31003| Fri Jan 18 13:12:48.007 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:48.007 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:48.007 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:48.097 [initandlisten] connection accepted from 127.0.0.5:55997 #252 (8 connections now open) m31002| Fri Jan 18 13:12:48.097 [conn251] end connection 127.0.0.5:45295 (6 connections now open) m31003| Fri Jan 18 13:12:48.367 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:48.367 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:48.367 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:48.444 [initandlisten] connection accepted from 127.0.0.5:57743 #253 (8 connections now open) m31002| Fri Jan 18 13:12:48.444 [conn252] end connection 127.0.0.5:55997 (6 connections now open) m31003| Fri Jan 18 13:12:48.611 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:48.611 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:48.611 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:48.698 [initandlisten] connection accepted from 127.0.0.5:58122 #254 (8 connections now open) m31002| Fri Jan 18 13:12:48.698 [conn253] end connection 127.0.0.5:57743 (7 connections now open) m31003| Fri Jan 18 13:12:48.834 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:48.834 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:48.834 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:48.895 [conn254] end connection 127.0.0.5:58122 (6 connections now open) m31002| Fri Jan 18 13:12:48.963 [initandlisten] connection accepted from 127.0.0.5:52899 #255 (7 connections now open) m31002| Fri Jan 18 13:12:49.042 [conn255] end connection 127.0.0.5:52899 (6 connections now open) m31003| Fri Jan 18 13:12:49.046 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:49.046 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:49.046 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:49.100 [initandlisten] connection accepted from 127.0.0.5:45490 #256 (7 connections now open) m31002| Fri Jan 18 13:12:49.241 [conn256] end connection 127.0.0.5:45490 (6 connections now open) m31003| Fri Jan 18 13:12:49.245 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:49.245 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:49.245 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:49.266 [initandlisten] connection accepted from 127.0.0.5:34534 #257 (7 connections now open) m31003| Fri Jan 18 13:12:49.413 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:49.413 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:49.413 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:49.486 [initandlisten] connection accepted from 127.0.0.5:39300 #258 (8 connections now open) m31002| Fri Jan 18 13:12:49.487 [conn257] end connection 127.0.0.5:34534 (7 connections now open) m31002| Fri Jan 18 13:12:49.636 [conn258] end connection 127.0.0.5:39300 (6 connections now open) m31003| Fri Jan 18 13:12:49.640 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:49.640 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:49.640 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:49.739 [initandlisten] connection accepted from 127.0.0.5:47987 #259 (7 connections now open) m31003| Fri Jan 18 13:12:49.872 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:49.872 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:49.872 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:49.892 [conn259] end connection 127.0.0.5:47987 (6 connections now open) m31002| Fri Jan 18 13:12:49.986 [initandlisten] connection accepted from 127.0.0.5:40326 #260 (7 connections now open) m31002| Fri Jan 18 13:12:50.231 [conn260] end connection 127.0.0.5:40326 (6 connections now open) m31003| Fri Jan 18 13:12:50.235 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:50.235 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:50.235 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:50.248 [initandlisten] connection accepted from 127.0.0.5:39449 #261 (7 connections now open) m31002| Fri Jan 18 13:12:50.464 [conn261] end connection 127.0.0.5:39449 (6 connections now open) m31003| Fri Jan 18 13:12:50.770 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:50.770 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:50.770 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31000| Fri Jan 18 13:12:50.785 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293837 146ms m31002| Fri Jan 18 13:12:50.835 [initandlisten] connection accepted from 127.0.0.5:41667 #262 (7 connections now open) m31003| Fri Jan 18 13:12:51.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:51.009 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:51.009 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:51.047 [conn262] end connection 127.0.0.5:41667 (6 connections now open) m31002| Fri Jan 18 13:12:51.106 [initandlisten] connection accepted from 127.0.0.5:43120 #263 (7 connections now open) m31003| Fri Jan 18 13:12:51.270 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:51.270 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:51.270 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:51.333 [conn263] end connection 127.0.0.5:43120 (6 connections now open) m31002| Fri Jan 18 13:12:51.333 [initandlisten] connection accepted from 127.0.0.5:43367 #264 (8 connections now open) m31003| Fri Jan 18 13:12:51.472 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:51.472 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:51.472 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:12:51.569 [conn264] end connection 127.0.0.5:43367 (6 connections now open) m31002| Fri Jan 18 13:12:51.593 [initandlisten] connection accepted from 127.0.0.5:56053 #265 (7 connections now open) m31003| Fri Jan 18 13:12:51.758 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:51.758 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:51.758 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:51.758 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:51.758 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Fri Jan 18 13:12:51.768 [conn265] end connection 127.0.0.5:56053 (6 connections now open) m31001| Fri Jan 18 13:12:51.844 [initandlisten] connection accepted from 127.0.0.5:60495 #236 (6 connections now open) m31003| Fri Jan 18 13:12:51.946 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31003| Fri Jan 18 13:12:51.946 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:51.946 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:51.946 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:51.946 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:12:51.956 [conn235] end connection 127.0.0.5:44611 (5 connections now open) m31001| Fri Jan 18 13:12:52.037 [initandlisten] connection accepted from 127.0.0.5:33033 #237 (7 connections now open) m31001| Fri Jan 18 13:12:52.037 [conn236] end connection 127.0.0.5:60495 (6 connections now open) m31001| Fri Jan 18 13:12:52.037 [initandlisten] connection accepted from 127.0.0.5:55278 #238 (8 connections now open) m31001| Fri Jan 18 13:12:52.201 [conn238] end connection 127.0.0.5:55278 (6 connections now open) m31003| Fri Jan 18 13:12:52.205 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:52.205 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:52.205 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:52.205 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:52.281 [initandlisten] connection accepted from 127.0.0.5:47647 #239 (7 connections now open) m31000| Fri Jan 18 13:12:52.360 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 numYields: 1211 locks(micros) r:285570 nreturned:154918 reslen:2633626 163ms m31003| Fri Jan 18 13:12:52.369 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:52.369 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:52.369 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:52.369 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31000| Fri Jan 18 13:12:52.421 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 numYields: 1211 locks(micros) r:281532 nreturned:154918 reslen:2633626 164ms m31001| Fri Jan 18 13:12:52.443 [initandlisten] connection accepted from 127.0.0.5:45729 #240 (8 connections now open) m31001| Fri Jan 18 13:12:52.443 [conn239] end connection 127.0.0.5:47647 (6 connections now open) m31001| Fri Jan 18 13:12:52.483 [conn237] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532595000|3565 } } cursorid:1109717021714243 ntoreturn:0 keyUpdates:0 numYields: 3 locks(micros) r:260476 nreturned:246722 reslen:4194294 201ms m31003| Fri Jan 18 13:12:52.589 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:52.589 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:52.589 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:52.589 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:52.666 [initandlisten] connection accepted from 127.0.0.5:59040 #241 (8 connections now open) m31001| Fri Jan 18 13:12:52.666 [conn240] end connection 127.0.0.5:45729 (6 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:52Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 262, "optime" : { "t" : 1358532772000, "i" : 6794 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532732000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532732000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532772000, "i" : 2366 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 227, "optime" : { "t" : 1358532771000, "i" : 4510 }, "optimeDate" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 262, "optime" : { "t" : 1358532732000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:12Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532595000, "i" : 3265 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:51Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532771000, "i" : 3944 }, "optimeDate" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:51Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 226, "optime" : { "t" : 1358532772000, "i" : 888 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 261, "optime" : { "t" : 1358532732000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:12Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358532598000, "i" : 100 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 1, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358532772000, "i" : 797 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 m31001| Fri Jan 18 13:12:52.830 [conn241] end connection 127.0.0.5:59040 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:52Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 16 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 26 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532598000, "i" : 300 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358532771000, "i" : 8428 }, "optimeDate" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:52.831 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:52.831 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:52.831 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:52.831 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:52Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 227, "optime" : { "t" : 1358532771000, "i" : 4836 }, "optimeDate" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532729000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:09Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:52Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358532595000, "i" : 3265 }, "optimeDate" : ISODate("2013-01-18T18:09:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:51Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532772000, "i" : 6794 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:12:52.836 [initandlisten] connection accepted from 127.0.0.5:39264 #242 (7 connections now open) m31003| Fri Jan 18 13:12:53.036 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:53.036 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:53.036 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:53.036 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:53.050 [initandlisten] connection accepted from 127.0.0.5:41671 #243 (8 connections now open) m31001| Fri Jan 18 13:12:53.050 [conn242] end connection 127.0.0.5:39264 (7 connections now open) m31003| Fri Jan 18 13:12:53.154 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:53.154 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:53.154 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:53.154 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:53.208 [initandlisten] connection accepted from 127.0.0.5:45697 #244 (8 connections now open) m31001| Fri Jan 18 13:12:53.208 [conn243] end connection 127.0.0.5:41671 (7 connections now open) m31003| Fri Jan 18 13:12:53.403 [conn29] end connection 127.0.0.3:47918 (3 connections now open) m31003| Fri Jan 18 13:12:53.403 [initandlisten] connection accepted from 127.0.0.3:46348 #32 (5 connections now open) m31002| Fri Jan 18 13:12:53.452 [conn175] end connection 127.0.0.6:57914 (4 connections now open) m31002| Fri Jan 18 13:12:53.452 [initandlisten] connection accepted from 127.0.0.6:56765 #266 (5 connections now open) m31003| Fri Jan 18 13:12:53.497 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:53.497 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:53.497 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:53.497 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:53.557 [conn244] end connection 127.0.0.5:45697 (6 connections now open) m31001| Fri Jan 18 13:12:53.558 [initandlisten] connection accepted from 127.0.0.5:56915 #245 (7 connections now open) m31003| Fri Jan 18 13:12:53.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:53.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:53.811 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:53.811 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:53.874 [initandlisten] connection accepted from 127.0.0.5:45037 #246 (8 connections now open) m31001| Fri Jan 18 13:12:53.874 [conn245] end connection 127.0.0.5:56915 (7 connections now open) m31001| Fri Jan 18 13:12:54.033 [conn246] end connection 127.0.0.5:45037 (6 connections now open) m31003| Fri Jan 18 13:12:54.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:54.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:54.037 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:54.037 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:54.087 [initandlisten] connection accepted from 127.0.0.5:53678 #247 (7 connections now open) m31003| Fri Jan 18 13:12:54.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:54.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:54.279 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:54.279 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:12:54.329 [conn30] end connection 127.0.0.4:57469 (3 connections now open) m31003| Fri Jan 18 13:12:54.330 [initandlisten] connection accepted from 127.0.0.4:60558 #33 (4 connections now open) m31001| Fri Jan 18 13:12:54.371 [initandlisten] connection accepted from 127.0.0.5:45581 #248 (8 connections now open) m31001| Fri Jan 18 13:12:54.372 [conn247] end connection 127.0.0.5:53678 (7 connections now open) m31001| Fri Jan 18 13:12:54.448 [conn196] end connection 127.0.0.2:44076 (6 connections now open) m31001| Fri Jan 18 13:12:54.449 [initandlisten] connection accepted from 127.0.0.2:60506 #249 (7 connections now open) m31001| Fri Jan 18 13:12:54.607 [conn248] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1000 } } cursorid:1119888063555481 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1189 locks(micros) r:207597 nreturned:101 reslen:10423 114ms m31001| Fri Jan 18 13:12:54.607 [conn248] end connection 127.0.0.5:45581 (6 connections now open) m31003| Fri Jan 18 13:12:54.609 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:54.609 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:54.609 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:54.609 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:54.610 [initandlisten] connection accepted from 127.0.0.5:36383 #250 (7 connections now open) m31001| Fri Jan 18 13:12:54.875 [conn250] end connection 127.0.0.5:36383 (6 connections now open) m31003| Fri Jan 18 13:12:54.877 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:54.877 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:54.877 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:54.877 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:55.060 [initandlisten] connection accepted from 127.0.0.5:53764 #251 (7 connections now open) m31001| Fri Jan 18 13:12:55.225 [conn251] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1200 } } cursorid:1122530170630593 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1225 locks(micros) r:212001 nreturned:101 reslen:10423 107ms m31001| Fri Jan 18 13:12:55.226 [conn251] end connection 127.0.0.5:53764 (6 connections now open) m31003| Fri Jan 18 13:12:55.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:55.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:55.230 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:55.230 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:55.230 [initandlisten] connection accepted from 127.0.0.5:46373 #252 (7 connections now open) m31001| Fri Jan 18 13:12:55.400 [conn252] end connection 127.0.0.5:46373 (6 connections now open) m31000| Fri Jan 18 13:12:55.403 [conn51] end connection 127.0.0.3:42313 (11 connections now open) m31000| Fri Jan 18 13:12:55.403 [initandlisten] connection accepted from 127.0.0.3:35493 #54 (12 connections now open) m31003| Fri Jan 18 13:12:55.423 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:55.423 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:55.423 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:55.423 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31000| Fri Jan 18 13:12:55.452 [conn52] end connection 127.0.0.6:49331 (11 connections now open) m31000| Fri Jan 18 13:12:55.452 [initandlisten] connection accepted from 127.0.0.6:56963 #55 (12 connections now open) m31001| Fri Jan 18 13:12:55.494 [initandlisten] connection accepted from 127.0.0.5:37867 #253 (7 connections now open) m31001| Fri Jan 18 13:12:55.656 [conn253] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1400 } } cursorid:1124371862466685 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1224 locks(micros) r:207491 nreturned:101 reslen:10423 105ms m31001| Fri Jan 18 13:12:55.656 [conn253] end connection 127.0.0.5:37867 (6 connections now open) m31003| Fri Jan 18 13:12:55.658 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:55.658 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:55.658 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:55.658 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31002| Fri Jan 18 13:12:55.807 [conn186] end connection 127.0.0.5:54722 (4 connections now open) m31002| Fri Jan 18 13:12:55.818 [initandlisten] connection accepted from 127.0.0.5:42564 #267 (5 connections now open) m31001| Fri Jan 18 13:12:55.823 [initandlisten] connection accepted from 127.0.0.5:60640 #254 (7 connections now open) m31003| Fri Jan 18 13:12:55.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Fri Jan 18 13:12:56.038 [conn254] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1500 } } cursorid:1125987182768182 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1237 locks(micros) r:203049 nreturned:101 reslen:10423 102ms m31001| Fri Jan 18 13:12:56.038 [conn254] end connection 127.0.0.5:60640 (6 connections now open) m31003| Fri Jan 18 13:12:56.040 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:56.040 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:56.040 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:56.040 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31000| Fri Jan 18 13:12:56.101 [conn53] end connection 127.0.0.4:40524 (11 connections now open) m31000| Fri Jan 18 13:12:56.102 [initandlisten] connection accepted from 127.0.0.4:34591 #56 (12 connections now open) m31001| Fri Jan 18 13:12:56.105 [initandlisten] connection accepted from 127.0.0.5:60813 #255 (7 connections now open) m31001| Fri Jan 18 13:12:56.239 [conn255] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1600 } } cursorid:1126840820975518 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1236 locks(micros) r:199663 nreturned:101 reslen:10423 101ms m31003| Fri Jan 18 13:12:56.242 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:56.242 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:56.242 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:56.242 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:56.310 [conn255] end connection 127.0.0.5:60813 (6 connections now open) m31001| Fri Jan 18 13:12:56.310 [initandlisten] connection accepted from 127.0.0.5:38811 #256 (8 connections now open) m31001| Fri Jan 18 13:12:56.502 [conn256] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1700 } } cursorid:1127970694584235 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1236 locks(micros) r:208728 nreturned:101 reslen:10423 106ms m31003| Fri Jan 18 13:12:56.504 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:56.504 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:56.504 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:56.504 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:56.596 [conn256] end connection 127.0.0.5:38811 (6 connections now open) m31001| Fri Jan 18 13:12:56.629 [initandlisten] connection accepted from 127.0.0.5:44998 #257 (7 connections now open) m31001| Fri Jan 18 13:12:56.780 [conn257] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|1800 } } cursorid:1129155992229444 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1236 locks(micros) r:201311 nreturned:101 reslen:10423 101ms m31003| Fri Jan 18 13:12:56.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:56.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:56.784 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:56.784 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:56.790 [conn257] end connection 127.0.0.5:44998 (6 connections now open) m31001| Fri Jan 18 13:12:56.886 [initandlisten] connection accepted from 127.0.0.5:53573 #258 (7 connections now open) m31003| Fri Jan 18 13:12:56.986 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:56.986 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:56.986 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:56.986 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:57.078 [conn258] end connection 127.0.0.5:53573 (6 connections now open) m31001| Fri Jan 18 13:12:57.173 [initandlisten] connection accepted from 127.0.0.5:58111 #259 (7 connections now open) m31001| Fri Jan 18 13:12:57.327 [conn259] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2000 } } cursorid:1131470666569065 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1235 locks(micros) r:200648 nreturned:101 reslen:10423 101ms m31001| Fri Jan 18 13:12:57.327 [conn259] end connection 127.0.0.5:58111 (6 connections now open) m31003| Fri Jan 18 13:12:57.329 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:57.329 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:57.329 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:57.329 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:57.361 [initandlisten] connection accepted from 127.0.0.5:38396 #260 (7 connections now open) m31003| Fri Jan 18 13:12:57.483 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:57.483 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:57.483 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:57.483 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:57.579 [conn260] end connection 127.0.0.5:38396 (6 connections now open) m31001| Fri Jan 18 13:12:57.591 [initandlisten] connection accepted from 127.0.0.5:42849 #261 (7 connections now open) m31001| Fri Jan 18 13:12:57.794 [conn261] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2200 } } cursorid:1133469343435569 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1234 locks(micros) r:200561 nreturned:101 reslen:10423 101ms m31003| Fri Jan 18 13:12:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:57.797 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:57.797 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:57.813 [conn261] end connection 127.0.0.5:42849 (6 connections now open) m31001| Fri Jan 18 13:12:57.840 [initandlisten] connection accepted from 127.0.0.5:51474 #262 (7 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 268, "optime" : { "t" : 1358532778000, "i" : 339 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532776000, "i" : 3995 }, "optimeDate" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 233, "optime" : { "t" : 1358532777000, "i" : 4660 }, "optimeDate" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 268, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532598000, "i" : 2100 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532777000, "i" : 4641 }, "optimeDate" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 232, "optime" : { "t" : 1358532776000, "i" : 844 }, "optimeDate" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:56Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 232, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 267, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 232, "optime" : { "t" : 1358532598000, "i" : 1700 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 232, "optime" : { "t" : 1358532776000, "i" : 844 }, "optimeDate" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:58Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 22 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 43 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532598000, "i" : 2300 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358532776000, "i" : 8115 }, "optimeDate" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:57Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:58.104 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:58.104 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:58.104 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:58.104 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:12:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 233, "optime" : { "t" : 1358532777000, "i" : 4855 }, "optimeDate" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532735000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:12:58Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358532598000, "i" : 2100 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532778000, "i" : 339 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:12:58.150 [conn262] end connection 127.0.0.5:51474 (6 connections now open) m31001| Fri Jan 18 13:12:58.170 [initandlisten] connection accepted from 127.0.0.5:50949 #263 (7 connections now open) m31003| Fri Jan 18 13:12:58.348 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:58.348 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:58.348 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:58.348 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:58.415 [conn263] end connection 127.0.0.5:50949 (6 connections now open) m31001| Fri Jan 18 13:12:58.522 [initandlisten] connection accepted from 127.0.0.5:35727 #264 (7 connections now open) m31001| Fri Jan 18 13:12:58.681 [conn264] end connection 127.0.0.5:35727 (6 connections now open) m31003| Fri Jan 18 13:12:58.684 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:58.684 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:58.684 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:58.684 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:58.684 [initandlisten] connection accepted from 127.0.0.5:34159 #265 (7 connections now open) m31001| Fri Jan 18 13:12:58.842 [conn265] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2600 } } cursorid:1137901380840023 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1232 locks(micros) r:214961 nreturned:101 reslen:10423 108ms m31003| Fri Jan 18 13:12:58.844 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:58.844 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:58.844 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:58.844 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:58.935 [initandlisten] connection accepted from 127.0.0.5:49703 #266 (8 connections now open) m31001| Fri Jan 18 13:12:58.935 [conn265] end connection 127.0.0.5:34159 (6 connections now open) m31001| Fri Jan 18 13:12:59.170 [conn266] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2700 } } cursorid:1139314482056275 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1259 locks(micros) r:216762 nreturned:101 reslen:10423 110ms m31001| Fri Jan 18 13:12:59.170 [conn266] end connection 127.0.0.5:49703 (6 connections now open) m31003| Fri Jan 18 13:12:59.174 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:59.174 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:59.174 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:59.174 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:59.279 [initandlisten] connection accepted from 127.0.0.5:51054 #267 (7 connections now open) m31001| Fri Jan 18 13:12:59.426 [conn267] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2800 } } cursorid:1140392409765523 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1258 locks(micros) r:239211 nreturned:101 reslen:10423 147ms m31003| Fri Jan 18 13:12:59.429 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:59.429 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:59.429 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:59.429 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:59.507 [conn267] end connection 127.0.0.5:51054 (6 connections now open) m31001| Fri Jan 18 13:12:59.510 [initandlisten] connection accepted from 127.0.0.5:49169 #268 (7 connections now open) m31001| Fri Jan 18 13:12:59.716 [conn268] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|2900 } } cursorid:1141633956121095 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1297 locks(micros) r:203621 nreturned:101 reslen:10423 103ms m31003| Fri Jan 18 13:12:59.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:59.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:59.720 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:59.720 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:59.764 [conn268] end connection 127.0.0.5:49169 (6 connections now open) m31001| Fri Jan 18 13:12:59.825 [initandlisten] connection accepted from 127.0.0.5:36944 #269 (7 connections now open) m31001| Fri Jan 18 13:12:59.983 [conn269] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3000 } } cursorid:1142754416587556 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1296 locks(micros) r:239307 nreturned:101 reslen:10423 122ms m31003| Fri Jan 18 13:12:59.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:12:59.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:12:59.985 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:12:59.985 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:12:59.995 [conn269] end connection 127.0.0.5:36944 (6 connections now open) m31001| Fri Jan 18 13:13:00.080 [initandlisten] connection accepted from 127.0.0.5:41348 #270 (7 connections now open) m31001| Fri Jan 18 13:13:00.281 [conn270] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3100 } } cursorid:1144013109719217 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1296 locks(micros) r:212861 nreturned:101 reslen:10423 107ms m31001| Fri Jan 18 13:13:00.281 [conn270] end connection 127.0.0.5:41348 (6 connections now open) m31003| Fri Jan 18 13:13:00.284 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:00.284 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:00.284 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:00.284 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:00.386 [initandlisten] connection accepted from 127.0.0.5:55786 #271 (7 connections now open) m31001| Fri Jan 18 13:13:00.489 [conn271] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3200 } } cursorid:1144926796403254 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1296 locks(micros) r:202143 nreturned:101 reslen:10423 102ms m31003| Fri Jan 18 13:13:00.493 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:00.493 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:00.493 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:00.493 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:00.543 [initandlisten] connection accepted from 127.0.0.5:34096 #272 (8 connections now open) m31001| Fri Jan 18 13:13:00.543 [conn271] end connection 127.0.0.5:55786 (7 connections now open) m31001| Fri Jan 18 13:13:00.733 [conn272] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3300 } } cursorid:1145936854065934 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1302 locks(micros) r:202614 nreturned:101 reslen:10423 102ms m31003| Fri Jan 18 13:13:00.737 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:00.737 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:00.737 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:00.737 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:00.810 [conn272] end connection 127.0.0.5:34096 (6 connections now open) m31001| Fri Jan 18 13:13:00.837 [initandlisten] connection accepted from 127.0.0.5:42629 #273 (7 connections now open) m31001| Fri Jan 18 13:13:00.979 [conn273] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3400 } } cursorid:1147015990618359 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1301 locks(micros) r:231769 nreturned:101 reslen:10423 117ms m31003| Fri Jan 18 13:13:00.984 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:00.984 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:00.984 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:00.984 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:01.067 [conn273] end connection 127.0.0.5:42629 (6 connections now open) m31001| Fri Jan 18 13:13:01.067 [initandlisten] connection accepted from 127.0.0.5:40963 #274 (8 connections now open) m31001| Fri Jan 18 13:13:01.221 [conn274] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3500 } } cursorid:1148033722734889 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1301 locks(micros) r:202804 nreturned:101 reslen:10423 102ms m31003| Fri Jan 18 13:13:01.223 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:01.223 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:01.223 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:01.223 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:01.260 [conn274] end connection 127.0.0.5:40963 (6 connections now open) m31001| Fri Jan 18 13:13:01.284 [initandlisten] connection accepted from 127.0.0.5:50721 #275 (7 connections now open) m31001| Fri Jan 18 13:13:01.475 [conn275] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3600 } } cursorid:1149081374115119 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1301 locks(micros) r:210787 nreturned:101 reslen:10423 106ms m31001| Fri Jan 18 13:13:01.475 [conn275] end connection 127.0.0.5:50721 (6 connections now open) m31003| Fri Jan 18 13:13:01.477 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:01.477 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:01.477 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:01.477 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:01.517 [initandlisten] connection accepted from 127.0.0.5:41034 #276 (7 connections now open) m31001| Fri Jan 18 13:13:01.619 [conn276] end connection 127.0.0.5:41034 (6 connections now open) m31003| Fri Jan 18 13:13:01.621 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:01.621 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:01.621 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:01.621 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:01.624 [initandlisten] connection accepted from 127.0.0.5:54668 #277 (7 connections now open) m31001| Fri Jan 18 13:13:01.857 [conn277] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3800 } } cursorid:1150739747557899 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1300 locks(micros) r:215178 nreturned:101 reslen:10423 109ms m31003| Fri Jan 18 13:13:01.859 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:01.859 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:01.859 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:01.859 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:01.890 [initandlisten] connection accepted from 127.0.0.5:59759 #278 (8 connections now open) m31001| Fri Jan 18 13:13:01.890 [conn277] end connection 127.0.0.5:54668 (6 connections now open) m31001| Fri Jan 18 13:13:02.165 [conn278] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|3900 } } cursorid:1152045260405983 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1299 locks(micros) r:212721 nreturned:101 reslen:10423 107ms m31001| Fri Jan 18 13:13:02.165 [conn278] end connection 127.0.0.5:59759 (6 connections now open) m31003| Fri Jan 18 13:13:02.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:02.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:02.167 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:02.167 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:02.218 [initandlisten] connection accepted from 127.0.0.5:56707 #279 (7 connections now open) m31001| Fri Jan 18 13:13:02.377 [conn279] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4000 } } cursorid:1152919996934377 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1299 locks(micros) r:210544 nreturned:101 reslen:10423 106ms m31003| Fri Jan 18 13:13:02.379 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:02.379 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:02.379 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:02.379 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:02.446 [initandlisten] connection accepted from 127.0.0.5:54706 #280 (8 connections now open) m31001| Fri Jan 18 13:13:02.446 [conn279] end connection 127.0.0.5:56707 (7 connections now open) m31001| Fri Jan 18 13:13:02.622 [conn280] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4100 } } cursorid:1153977272902758 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1299 locks(micros) r:200407 nreturned:101 reslen:10423 101ms m31003| Fri Jan 18 13:13:02.624 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:02.624 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:02.624 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:02.624 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:02.632 [conn280] end connection 127.0.0.5:54706 (6 connections now open) m31001| Fri Jan 18 13:13:02.692 [initandlisten] connection accepted from 127.0.0.5:39246 #281 (7 connections now open) m31001| Fri Jan 18 13:13:02.833 [conn281] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4200 } } cursorid:1154852953274448 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1298 locks(micros) r:212548 nreturned:101 reslen:10423 107ms m31003| Fri Jan 18 13:13:02.835 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:02.835 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:02.835 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:02.835 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:02.915 [conn281] end connection 127.0.0.5:39246 (6 connections now open) m31001| Fri Jan 18 13:13:02.980 [initandlisten] connection accepted from 127.0.0.5:45868 #282 (7 connections now open) m31001| Fri Jan 18 13:13:03.090 [conn282] end connection 127.0.0.5:45868 (6 connections now open) m31003| Fri Jan 18 13:13:03.093 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:03.093 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:03.093 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:03.093 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:03.229 [initandlisten] connection accepted from 127.0.0.5:49009 #283 (7 connections now open) m31001| Fri Jan 18 13:13:03.364 [conn283] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4400 } } cursorid:1157137234279565 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1297 locks(micros) r:210514 nreturned:101 reslen:10423 106ms m31003| Fri Jan 18 13:13:03.366 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:03.366 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:03.366 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:03.366 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 273, "optime" : { "t" : 1358532783000, "i" : 3390 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532740000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532740000, "i" : 768 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532782000, "i" : 3773 }, "optimeDate" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 238, "optime" : { "t" : 1358532781000, "i" : 2346 }, "optimeDate" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 273, "optime" : { "t" : 1358532740000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532740000, "i" : 768 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532598000, "i" : 3600 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532781000, "i" : 2337 }, "optimeDate" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:01Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 237, "optime" : { "t" : 1358532782000, "i" : 446 }, "optimeDate" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358532740000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 272, "optime" : { "t" : 1358532740000, "i" : 768 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358532598000, "i" : 4000 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 237, "optime" : { "t" : 1358532782000, "i" : 440 }, "optimeDate" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:03Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532740000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 22 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532740000, "i" : 768 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 27 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532598000, "i" : 4500 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358532781000, "i" : 6488 }, "optimeDate" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 238, "optime" : { "t" : 1358532781000, "i" : 3001 }, "optimeDate" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532740000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532740000, "i" : 768 }, "optimeDate" : ISODate("2013-01-18T18:12:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:02Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358532598000, "i" : 3600 }, "optimeDate" : ISODate("2013-01-18T18:09:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532783000, "i" : 2791 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.3:31001" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:13:03.436 [initandlisten] connection accepted from 127.0.0.5:58226 #284 (8 connections now open) m31001| Fri Jan 18 13:13:03.436 [conn283] end connection 127.0.0.5:49009 (6 connections now open) m31001| Fri Jan 18 13:13:03.540 [conn284] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4500 } } cursorid:1157890339519323 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1297 locks(micros) r:203036 nreturned:101 reslen:10423 102ms m31003| Fri Jan 18 13:13:03.542 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:03.542 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:03.543 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:03.543 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:03.636 [conn284] end connection 127.0.0.5:58226 (6 connections now open) m31001| Fri Jan 18 13:13:03.640 [initandlisten] connection accepted from 127.0.0.5:41596 #285 (7 connections now open) m31001| Fri Jan 18 13:13:03.898 [conn285] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4600 } } cursorid:1159397269846386 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1310 locks(micros) r:214404 nreturned:101 reslen:10423 109ms m31003| Fri Jan 18 13:13:03.902 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:03.903 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:03.903 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:03.903 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31001| Fri Jan 18 13:13:03.975 [conn285] end connection 127.0.0.5:41596 (6 connections now open) m31001| Fri Jan 18 13:13:03.975 [initandlisten] connection accepted from 127.0.0.5:35286 #286 (8 connections now open) m31001| Fri Jan 18 13:13:04.150 [conn286] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532598000|4700 } } cursorid:1160479600602345 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1329 locks(micros) r:223969 nreturned:101 reslen:10423 113ms m31003| Fri Jan 18 13:13:04.153 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:04.153 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:04.153 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31003| Fri Jan 18 13:13:04.153 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31002| Fri Jan 18 13:13:04.153 [initandlisten] connection accepted from 127.0.0.5:59774 #268 (6 connections now open) m31001| Fri Jan 18 13:13:04.158 [conn286] end connection 127.0.0.5:35286 (6 connections now open) m31002| Fri Jan 18 13:13:04.359 [conn268] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532599000|55 } } cursorid:1158503286007840 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1328 locks(micros) r:223096 nreturned:101 reslen:10423 112ms m31003| Fri Jan 18 13:13:04.363 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31003| Fri Jan 18 13:13:04.363 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:04.363 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:04.363 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:13:04.388 [conn268] end connection 127.0.0.5:59774 (5 connections now open) m31001| Fri Jan 18 13:13:04.406 [conn237] end connection 127.0.0.5:33033 (5 connections now open) m31000| Fri Jan 18 13:13:04.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Fri Jan 18 13:13:04.442 [initandlisten] connection accepted from 127.0.0.5:57311 #269 (6 connections now open) m31002| Fri Jan 18 13:13:04.442 [initandlisten] connection accepted from 127.0.0.5:50917 #270 (7 connections now open) m31002| Fri Jan 18 13:13:04.615 [conn269] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532599000|155 } } cursorid:1159607223607700 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1328 locks(micros) r:215999 nreturned:101 reslen:1737 109ms m31002| Fri Jan 18 13:13:04.704 [conn270] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532599000|155 } } cursorid:1159950544208504 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1328 locks(micros) r:205540 nreturned:101 reslen:10423 104ms m31003| Fri Jan 18 13:13:04.708 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:04.708 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:04.708 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:13:04.750 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.2, filling with zeroes... m31002| Fri Jan 18 13:13:04.787 [conn270] end connection 127.0.0.5:50917 (6 connections now open) m31002| Fri Jan 18 13:13:04.792 [initandlisten] connection accepted from 127.0.0.5:60114 #271 (7 connections now open) m31001| Fri Jan 18 13:13:04.925 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.2, filling with zeroes... m31002| Fri Jan 18 13:13:05.387 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.2, size: 64MB, took 0.636 secs m31001| Fri Jan 18 13:13:05.446 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.2, size: 64MB, took 0.52 secs m31002| Fri Jan 18 13:13:05.638 [conn271] query local.oplog.rs query: { ts: { $gte: Timestamp 1358532599000|255 } } cursorid:1163931954483216 ntoreturn:0 ntoskip:0 nscanned:103 keyUpdates:0 numYields: 1330 locks(micros) r:215497 nreturned:101 reslen:10423 242ms m31003| Fri Jan 18 13:13:05.640 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:05.640 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.4:31002 m31003| Fri Jan 18 13:13:05.640 [rsBackgroundSync] SYNCED TO 127.0.0.4:31002 m31002| Fri Jan 18 13:13:05.693 [conn271] end connection 127.0.0.5:60114 (6 connections now open) m31002| Fri Jan 18 13:13:05.707 [conn269] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532599000|155 } } cursorid:1159607223607700 ntoreturn:0 keyUpdates:0 numYields: 612 locks(micros) r:407650 nreturned:246722 reslen:4194294 319ms m31002| Fri Jan 18 13:13:05.774 [initandlisten] connection accepted from 127.0.0.5:35975 #272 (7 connections now open) m31003| Fri Jan 18 13:13:05.793 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:05.793 [rsBackgroundSync] replSet not trying to sync from 127.0.0.4:31002, it is vetoed for 600 more seconds m31003| Fri Jan 18 13:13:05.793 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.3:31001 m31003| Fri Jan 18 13:13:05.793 [rsBackgroundSync] SYNCED TO 127.0.0.3:31001 m31003| Fri Jan 18 13:13:05.793 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Fri Jan 18 13:13:05.793 [initandlisten] connection accepted from 127.0.0.5:42314 #287 (6 connections now open) m31003| Fri Jan 18 13:13:05.825 [rsBackgroundSync] CLOSEST CANDIDATE IS : 127.0.0.6:31004 m31003| Fri Jan 18 13:13:05.825 [rsBackgroundSync] replSet not trying to sync from 127.0.0.4:31002, it is vetoed for 600 more seconds m31003| Fri Jan 18 13:13:05.825 [rsBackgroundSync] replSet not trying to sync from 127.0.0.3:31001, it is vetoed for 600 more seconds m31003| Fri Jan 18 13:13:05.825 [rsBackgroundSync] SYNCED TO 127.0.0.6:31004 m31003| Fri Jan 18 13:13:05.825 [rsBackgroundSync] replSet syncing to: 127.0.0.6:31004 m31002| Fri Jan 18 13:13:05.872 [conn272] end connection 127.0.0.5:35975 (6 connections now open) m31001| Fri Jan 18 13:13:05.893 [conn287] end connection 127.0.0.5:42314 (5 connections now open) m31004| Fri Jan 18 13:13:06.450 [conn33] end connection 127.0.0.2:58263 (4 connections now open) m31004| Fri Jan 18 13:13:06.451 [initandlisten] connection accepted from 127.0.0.2:43519 #37 (5 connections now open) m31004| Fri Jan 18 13:13:06.826 [initandlisten] connection accepted from 127.0.0.5:36752 #38 (6 connections now open) m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet error RS102 too stale to catch up, at least from 127.0.0.6:31004 m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet our last optime : Jan 18 13:09:59 50f98ff7:163 m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet oldest at 127.0.0.6:31004 : Jan 18 13:11:31 50f99053:206f m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet error RS102 too stale to catch up m31003| Fri Jan 18 13:13:06.826 [rsBackgroundSync] replSet RECOVERING m31004| Fri Jan 18 13:13:07.326 [conn38] end connection 127.0.0.5:36752 (5 connections now open) m31004| Fri Jan 18 13:13:07.404 [conn34] end connection 127.0.0.3:49490 (4 connections now open) m31001| Fri Jan 18 13:13:07.404 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31004| Fri Jan 18 13:13:07.404 [initandlisten] connection accepted from 127.0.0.3:51349 #39 (5 connections now open) m31003| Fri Jan 18 13:13:07.454 [conn31] end connection 127.0.0.6:56478 (3 connections now open) m31003| Fri Jan 18 13:13:07.454 [initandlisten] connection accepted from 127.0.0.6:48523 #34 (4 connections now open) m31004| Fri Jan 18 13:13:07.454 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31003| Fri Jan 18 13:13:07.640 [rsSync] replSet still syncing, not yet to minValid optime 50f99053:206f m31004| Fri Jan 18 13:13:08.103 [conn35] end connection 127.0.0.4:39764 (4 connections now open) m31004| Fri Jan 18 13:13:08.103 [initandlisten] connection accepted from 127.0.0.4:35188 #40 (5 connections now open) m31002| Fri Jan 18 13:13:08.332 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31002| Fri Jan 18 13:13:08.450 [conn218] end connection 127.0.0.2:53080 (5 connections now open) m31002| Fri Jan 18 13:13:08.450 [initandlisten] connection accepted from 127.0.0.2:35177 #273 (6 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 278, "optime" : { "t" : 1358532788000, "i" : 5369 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532788000, "i" : 2894 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 243, "optime" : { "t" : 1358532787000, "i" : 3594 }, "optimeDate" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 278, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 243, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0, "lastHeartbeatMessage" : "error RS102 too stale to catch up" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358532787000, "i" : 3257 }, "optimeDate" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:07Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 242, "optime" : { "t" : 1358532788000, "i" : 477 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 242, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 277, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 242, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 242, "optime" : { "t" : 1358532788000, "i" : 429 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:08Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 18 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 276, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358532786000, "i" : 4369 }, "optimeDate" : ISODate("2013-01-18T18:13:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:07Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 243, "optime" : { "t" : 1358532787000, "i" : 4327 }, "optimeDate" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358532745000, "i" : 3134 }, "optimeDate" : ISODate("2013-01-18T18:12:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:08Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 243, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "error RS102 too stale to catch up" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532788000, "i" : 5369 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:13:09.405 [conn222] end connection 127.0.0.3:47540 (5 connections now open) m31002| Fri Jan 18 13:13:09.405 [initandlisten] connection accepted from 127.0.0.3:41193 #274 (6 connections now open) m31001| Fri Jan 18 13:13:09.454 [conn225] end connection 127.0.0.6:40655 (4 connections now open) m31001| Fri Jan 18 13:13:09.454 [initandlisten] connection accepted from 127.0.0.6:42600 #288 (5 connections now open) m31001| Fri Jan 18 13:13:10.103 [conn226] end connection 127.0.0.4:38759 (4 connections now open) m31001| Fri Jan 18 13:13:10.104 [initandlisten] connection accepted from 127.0.0.4:33582 #289 (5 connections now open) m31001| Fri Jan 18 13:13:12.079 [conn227] end connection 127.0.0.5:35520 (4 connections now open) m31001| Fri Jan 18 13:13:12.220 [initandlisten] connection accepted from 127.0.0.5:46064 #290 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 283, "optime" : { "t" : 1358532793000, "i" : 8397 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532792000, "i" : 3851 }, "optimeDate" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 248, "optime" : { "t" : 1358532793000, "i" : 3442 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 283, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 248, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358532793000, "i" : 2939 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 247, "optime" : { "t" : 1358532792000, "i" : 583 }, "optimeDate" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:12Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 247, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 282, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 247, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 247, "optime" : { "t" : 1358532792000, "i" : 172 }, "optimeDate" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:13Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 35 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:12Z"), "pingMs" : 18 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 281, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358532791000, "i" : 1576 }, "optimeDate" : ISODate("2013-01-18T18:13:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:13Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:13Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 248, "optime" : { "t" : 1358532793000, "i" : 4242 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358532751000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 248, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532793000, "i" : 8397 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:13:14.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Fri Jan 18 13:13:17.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 289, "optime" : { "t" : 1358532799000, "i" : 2749 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:12:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532798000, "i" : 3567 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 254, "optime" : { "t" : 1358532797000, "i" : 3595 }, "optimeDate" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 289, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 254, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358532797000, "i" : 3584 }, "optimeDate" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:17Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 253, "optime" : { "t" : 1358532798000, "i" : 847 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 288, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 253, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358532798000, "i" : 842 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:19Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 22 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 28 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 287, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 255, "optime" : { "t" : 1358532793000, "i" : 8397 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:17Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 254, "optime" : { "t" : 1358532797000, "i" : 3685 }, "optimeDate" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358532756000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:18Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 254, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:17Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532799000, "i" : 2749 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31004| Fri Jan 18 13:13:21.888 [conn36] end connection 127.0.0.5:37741 (4 connections now open) m31004| Fri Jan 18 13:13:22.388 [initandlisten] connection accepted from 127.0.0.5:51308 #41 (5 connections now open) m31003| Fri Jan 18 13:13:23.407 [conn32] end connection 127.0.0.3:46348 (3 connections now open) m31003| Fri Jan 18 13:13:23.407 [initandlisten] connection accepted from 127.0.0.3:52616 #35 (4 connections now open) m31002| Fri Jan 18 13:13:23.456 [conn266] end connection 127.0.0.6:56765 (5 connections now open) m31002| Fri Jan 18 13:13:23.456 [initandlisten] connection accepted from 127.0.0.6:59328 #275 (6 connections now open) m31003| Fri Jan 18 13:13:24.334 [conn33] end connection 127.0.0.4:60558 (3 connections now open) m31003| Fri Jan 18 13:13:24.334 [initandlisten] connection accepted from 127.0.0.4:41001 #36 (4 connections now open) m31001| Fri Jan 18 13:13:24.453 [conn249] end connection 127.0.0.2:60506 (4 connections now open) m31001| Fri Jan 18 13:13:24.453 [initandlisten] connection accepted from 127.0.0.2:40874 #291 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 294, "optime" : { "t" : 1358532804000, "i" : 5817 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532804000, "i" : 3441 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 259, "optime" : { "t" : 1358532803000, "i" : 3624 }, "optimeDate" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 294, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 259, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358532803000, "i" : 3409 }, "optimeDate" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:23Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 258, "optime" : { "t" : 1358532804000, "i" : 660 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 293, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 258, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358532804000, "i" : 621 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:24Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 50 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 25 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 292, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 260, "optime" : { "t" : 1358532802000, "i" : 360 }, "optimeDate" : ISODate("2013-01-18T18:13:22Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:23Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 259, "optime" : { "t" : 1358532803000, "i" : 4043 }, "optimeDate" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358532762000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:24Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 259, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532804000, "i" : 5817 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:13:25.407 [conn54] end connection 127.0.0.3:35493 (11 connections now open) m31000| Fri Jan 18 13:13:25.407 [initandlisten] connection accepted from 127.0.0.3:34644 #57 (12 connections now open) m31000| Fri Jan 18 13:13:25.456 [conn55] end connection 127.0.0.6:56963 (11 connections now open) m31000| Fri Jan 18 13:13:25.456 [initandlisten] connection accepted from 127.0.0.6:43976 #58 (12 connections now open) m31000| Fri Jan 18 13:13:26.106 [conn56] end connection 127.0.0.4:34591 (11 connections now open) m31000| Fri Jan 18 13:13:26.106 [initandlisten] connection accepted from 127.0.0.4:56886 #59 (13 connections now open) m31002| Fri Jan 18 13:13:26.248 [conn267] end connection 127.0.0.5:42564 (5 connections now open) m31002| Fri Jan 18 13:13:26.342 [initandlisten] connection accepted from 127.0.0.5:53682 #276 (6 connections now open) m31003| Fri Jan 18 13:13:27.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 300, "optime" : { "t" : 1358532810000, "i" : 169 }, "optimeDate" : ISODate("2013-01-18T18:13:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532808000, "i" : 3370 }, "optimeDate" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 265, "optime" : { "t" : 1358532809000, "i" : 3077 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 300, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 265, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 265, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 265, "optime" : { "t" : 1358532809000, "i" : 2942 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 264, "optime" : { "t" : 1358532808000, "i" : 579 }, "optimeDate" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 264, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 299, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 264, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:28Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 264, "optime" : { "t" : 1358532808000, "i" : 431 }, "optimeDate" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:30Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 32 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 30 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 298, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 266, "optime" : { "t" : 1358532809000, "i" : 7317 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 265, "optime" : { "t" : 1358532809000, "i" : 3887 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 265, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 265, "optime" : { "t" : 1358532767000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:12:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 265, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 298, "optime" : { "t" : 1358532810000, "i" : 169 }, "optimeDate" : ISODate("2013-01-18T18:13:30Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:13:32.940 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 locks(micros) r:124075 nreturned:145082 reslen:2466414 124ms m31000| Fri Jan 18 13:13:33.029 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 locks(micros) r:116084 nreturned:145082 reslen:2466414 116ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 305, "optime" : { "t" : 1358532815000, "i" : 2965 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532814000, "i" : 3922 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 270, "optime" : { "t" : 1358532813000, "i" : 2740 }, "optimeDate" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 305, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 270, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 270, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 270, "optime" : { "t" : 1358532813000, "i" : 2726 }, "optimeDate" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:33Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 269, "optime" : { "t" : 1358532814000, "i" : 1119 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 269, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 304, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 269, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 269, "optime" : { "t" : 1358532814000, "i" : 29 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:35Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 16 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 39 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 303, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 271, "optime" : { "t" : 1358532814000, "i" : 7661 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 270, "optime" : { "t" : 1358532813000, "i" : 2794 }, "optimeDate" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 270, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 270, "optime" : { "t" : 1358532772000, "i" : 5516 }, "optimeDate" : ISODate("2013-01-18T18:12:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 270, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:34Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 303, "optime" : { "t" : 1358532815000, "i" : 2411 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:13:36.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31004| Fri Jan 18 13:13:36.455 [conn37] end connection 127.0.0.2:43519 (4 connections now open) m31004| Fri Jan 18 13:13:36.455 [initandlisten] connection accepted from 127.0.0.2:55817 #42 (5 connections now open) m31004| Fri Jan 18 13:13:37.409 [conn39] end connection 127.0.0.3:51349 (4 connections now open) m31004| Fri Jan 18 13:13:37.409 [initandlisten] connection accepted from 127.0.0.3:40068 #43 (5 connections now open) m31003| Fri Jan 18 13:13:37.458 [conn34] end connection 127.0.0.6:48523 (3 connections now open) m31003| Fri Jan 18 13:13:37.458 [initandlisten] connection accepted from 127.0.0.6:43315 #37 (5 connections now open) m31004| Fri Jan 18 13:13:38.108 [conn40] end connection 127.0.0.4:35188 (4 connections now open) m31004| Fri Jan 18 13:13:38.108 [initandlisten] connection accepted from 127.0.0.4:46503 #44 (5 connections now open) m31002| Fri Jan 18 13:13:38.455 [conn273] end connection 127.0.0.2:35177 (5 connections now open) m31002| Fri Jan 18 13:13:38.455 [initandlisten] connection accepted from 127.0.0.2:48695 #277 (6 connections now open) m31002| Fri Jan 18 13:13:39.409 [conn274] end connection 127.0.0.3:41193 (5 connections now open) m31002| Fri Jan 18 13:13:39.409 [initandlisten] connection accepted from 127.0.0.3:39886 #278 (6 connections now open) m31001| Fri Jan 18 13:13:39.458 [conn288] end connection 127.0.0.6:42600 (4 connections now open) m31001| Fri Jan 18 13:13:39.458 [initandlisten] connection accepted from 127.0.0.6:58963 #292 (5 connections now open) m31001| Fri Jan 18 13:13:40.108 [conn289] end connection 127.0.0.4:33582 (4 connections now open) m31001| Fri Jan 18 13:13:40.108 [initandlisten] connection accepted from 127.0.0.4:57532 #293 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:40Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 310, "optime" : { "t" : 1358532820000, "i" : 6429 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532820000, "i" : 3582 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 275, "optime" : { "t" : 1358532819000, "i" : 3008 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 310, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 275, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 275, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 275, "optime" : { "t" : 1358532819000, "i" : 2987 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:39Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 274, "optime" : { "t" : 1358532820000, "i" : 1088 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 274, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 309, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 274, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 274, "optime" : { "t" : 1358532820000, "i" : 1072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:40Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 12 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 308, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 276, "optime" : { "t" : 1358532819000, "i" : 7358 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 275, "optime" : { "t" : 1358532819000, "i" : 3785 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 275, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 275, "optime" : { "t" : 1358532778000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:12:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:40Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 275, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:39Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 308, "optime" : { "t" : 1358532820000, "i" : 6429 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:13:42.391 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Fri Jan 18 13:13:42.618 [conn290] end connection 127.0.0.5:46064 (4 connections now open) m31001| Fri Jan 18 13:13:42.618 [initandlisten] connection accepted from 127.0.0.5:52697 #294 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:45Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 315, "optime" : { "t" : 1358532825000, "i" : 6825 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532783000, "i" : 1081 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532783000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532824000, "i" : 4378 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 280, "optime" : { "t" : 1358532825000, "i" : 3530 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 315, "optime" : { "t" : 1358532783000, "i" : 1081 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 280, "optime" : { "t" : 1358532783000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 280, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 280, "optime" : { "t" : 1358532825000, "i" : 3521 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 279, "optime" : { "t" : 1358532824000, "i" : 715 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:44Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 279, "optime" : { "t" : 1358532783000, "i" : 1081 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 314, "optime" : { "t" : 1358532783000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 279, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:44Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 279, "optime" : { "t" : 1358532824000, "i" : 704 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:45Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532783000, "i" : 1081 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 7 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532783000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 313, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 281, "optime" : { "t" : 1358532824000, "i" : 8553 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:45Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 280, "optime" : { "t" : 1358532825000, "i" : 4179 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 280, "optime" : { "t" : 1358532783000, "i" : 1081 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 280, "optime" : { "t" : 1358532783000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:03Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 280, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 313, "optime" : { "t" : 1358532825000, "i" : 6275 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:13:46.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Fri Jan 18 13:13:49.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 321, "optime" : { "t" : 1358532831000, "i" : 991 }, "optimeDate" : ISODate("2013-01-18T18:13:51Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532830000, "i" : 4109 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 286, "optime" : { "t" : 1358532829000, "i" : 3108 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 321, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 286, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 286, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 286, "optime" : { "t" : 1358532829000, "i" : 3100 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:49Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 285, "optime" : { "t" : 1358532830000, "i" : 999 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 285, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 320, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 285, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 285, "optime" : { "t" : 1358532830000, "i" : 993 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:51Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 8 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 43 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 319, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 287, "optime" : { "t" : 1358532829000, "i" : 7133 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:51Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 286, "optime" : { "t" : 1358532829000, "i" : 3912 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 286, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 286, "optime" : { "t" : 1358532788000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:08Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:50Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 286, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 319, "optime" : { "t" : 1358532831000, "i" : 991 }, "optimeDate" : ISODate("2013-01-18T18:13:51Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:13:53.411 [conn35] end connection 127.0.0.3:52616 (3 connections now open) m31003| Fri Jan 18 13:13:53.411 [initandlisten] connection accepted from 127.0.0.3:59985 #38 (4 connections now open) m31002| Fri Jan 18 13:13:53.460 [conn275] end connection 127.0.0.6:59328 (5 connections now open) m31002| Fri Jan 18 13:13:53.460 [initandlisten] connection accepted from 127.0.0.6:51458 #279 (6 connections now open) m31003| Fri Jan 18 13:13:54.339 [conn36] end connection 127.0.0.4:41001 (3 connections now open) m31003| Fri Jan 18 13:13:54.339 [initandlisten] connection accepted from 127.0.0.4:42305 #39 (4 connections now open) m31001| Fri Jan 18 13:13:54.457 [conn291] end connection 127.0.0.2:40874 (4 connections now open) m31001| Fri Jan 18 13:13:54.457 [initandlisten] connection accepted from 127.0.0.2:55557 #295 (5 connections now open) m31000| Fri Jan 18 13:13:55.411 [conn57] end connection 127.0.0.3:34644 (11 connections now open) m31000| Fri Jan 18 13:13:55.411 [initandlisten] connection accepted from 127.0.0.3:38425 #60 (12 connections now open) m31000| Fri Jan 18 13:13:55.460 [conn58] end connection 127.0.0.6:43976 (11 connections now open) m31000| Fri Jan 18 13:13:55.460 [initandlisten] connection accepted from 127.0.0.6:42467 #61 (12 connections now open) m31000| Fri Jan 18 13:13:55.693 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:626471 313ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:56Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 326, "optime" : { "t" : 1358532836000, "i" : 2245 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532793000, "i" : 6290 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532793000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532834000, "i" : 2856 }, "optimeDate" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 291, "optime" : { "t" : 1358532835000, "i" : 3716 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 326, "optime" : { "t" : 1358532793000, "i" : 6290 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 291, "optime" : { "t" : 1358532793000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 291, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 291, "optime" : { "t" : 1358532835000, "i" : 2242 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 m31000| Fri Jan 18 13:13:56.110 [conn59] end connection 127.0.0.4:56886 (11 connections now open) m31000| Fri Jan 18 13:13:56.110 [initandlisten] connection accepted from 127.0.0.4:43607 #62 (12 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 290, "optime" : { "t" : 1358532834000, "i" : 285 }, "optimeDate" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 290, "optime" : { "t" : 1358532793000, "i" : 6290 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 325, "optime" : { "t" : 1358532793000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 290, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:54Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 290, "optime" : { "t" : 1358532834000, "i" : 276 }, "optimeDate" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:56Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532793000, "i" : 6290 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 26 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532793000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 49 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 324, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 292, "optime" : { "t" : 1358532834000, "i" : 6704 }, "optimeDate" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:55Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:13:56Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 291, "optime" : { "t" : 1358532835000, "i" : 3716 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 291, "optime" : { "t" : 1358532793000, "i" : 6290 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 291, "optime" : { "t" : 1358532793000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:13Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:56Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 291, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 324, "optime" : { "t" : 1358532836000, "i" : 2205 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:13:57.029 [conn276] end connection 127.0.0.5:53682 (5 connections now open) m31002| Fri Jan 18 13:13:57.029 [initandlisten] connection accepted from 127.0.0.5:45716 #280 (7 connections now open) m31003| Fri Jan 18 13:13:59.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31004| Fri Jan 18 13:13:59.892 [conn41] end connection 127.0.0.5:51308 (4 connections now open) m31004| Fri Jan 18 13:14:00.393 [initandlisten] connection accepted from 127.0.0.5:33355 #45 (5 connections now open) m31000| Fri Jan 18 13:14:00.779 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:843376 421ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:01Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 331, "optime" : { "t" : 1358532841000, "i" : 4328 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532840000, "i" : 2365 }, "optimeDate" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 296, "optime" : { "t" : 1358532839000, "i" : 2529 }, "optimeDate" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 331, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 296, "optime" : { "t" : 1358532798000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 296, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 296, "optime" : { "t" : 1358532839000, "i" : 2512 }, "optimeDate" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:13:59Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 295, "optime" : { "t" : 1358532840000, "i" : 383 }, "optimeDate" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 295, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 330, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 295, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:01Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 295, "optime" : { "t" : 1358532840000, "i" : 360 }, "optimeDate" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:01Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 35 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532799000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:19Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 329, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 297, "optime" : { "t" : 1358532840000, "i" : 2365 }, "optimeDate" : ISODate("2013-01-18T18:14:00Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:01Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 296, "optime" : { "t" : 1358532839000, "i" : 2878 }, "optimeDate" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 296, "optime" : { "t" : 1358532798000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 296, "optime" : { "t" : 1358532798000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:18Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 296, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 329, "optime" : { "t" : 1358532841000, "i" : 4328 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:14:05.757 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:777111 388ms m31004| Fri Jan 18 13:14:06.459 [conn42] end connection 127.0.0.2:55817 (4 connections now open) m31004| Fri Jan 18 13:14:06.459 [initandlisten] connection accepted from 127.0.0.2:51891 #46 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:06Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 336, "optime" : { "t" : 1358532846000, "i" : 6891 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532804000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532804000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532846000, "i" : 6763 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 301, "optime" : { "t" : 1358532845000, "i" : 3051 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 336, "optime" : { "t" : 1358532804000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 301, "optime" : { "t" : 1358532804000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 301, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 301, "optime" : { "t" : 1358532845000, "i" : 2416 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:05Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 300, "optime" : { "t" : 1358532846000, "i" : 880 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 300, "optime" : { "t" : 1358532804000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 335, "optime" : { "t" : 1358532804000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 300, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 300, "optime" : { "t" : 1358532846000, "i" : 866 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:06Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532804000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:05Z"), "pingMs" : 22 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532804000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 36 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 334, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1358532845000, "i" : 2416 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:05Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:06Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 301, "optime" : { "t" : 1358532845000, "i" : 3051 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 301, "optime" : { "t" : 1358532804000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 301, "optime" : { "t" : 1358532804000, "i" : 1271 }, "optimeDate" : ISODate("2013-01-18T18:13:24Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:06Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 301, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 334, "optime" : { "t" : 1358532846000, "i" : 6891 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31004| Fri Jan 18 13:14:07.413 [conn43] end connection 127.0.0.3:40068 (4 connections now open) m31004| Fri Jan 18 13:14:07.413 [initandlisten] connection accepted from 127.0.0.3:58468 #47 (5 connections now open) m31003| Fri Jan 18 13:14:07.462 [conn37] end connection 127.0.0.6:43315 (3 connections now open) m31003| Fri Jan 18 13:14:07.462 [initandlisten] connection accepted from 127.0.0.6:48705 #40 (4 connections now open) m31003| Fri Jan 18 13:14:07.648 [rsSync] replSet still syncing, not yet to minValid optime 50f99053:206f m31004| Fri Jan 18 13:14:08.112 [conn44] end connection 127.0.0.4:46503 (4 connections now open) m31004| Fri Jan 18 13:14:08.113 [initandlisten] connection accepted from 127.0.0.4:47798 #48 (5 connections now open) m31000| Fri Jan 18 13:14:08.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Fri Jan 18 13:14:08.459 [conn277] end connection 127.0.0.2:48695 (5 connections now open) m31002| Fri Jan 18 13:14:08.459 [initandlisten] connection accepted from 127.0.0.2:57709 #281 (6 connections now open) m31002| Fri Jan 18 13:14:09.413 [conn278] end connection 127.0.0.3:39886 (5 connections now open) m31002| Fri Jan 18 13:14:09.413 [initandlisten] connection accepted from 127.0.0.3:55167 #282 (6 connections now open) m31001| Fri Jan 18 13:14:09.463 [conn292] end connection 127.0.0.6:58963 (4 connections now open) m31001| Fri Jan 18 13:14:09.463 [initandlisten] connection accepted from 127.0.0.6:55263 #296 (5 connections now open) m31001| Fri Jan 18 13:14:10.113 [conn293] end connection 127.0.0.4:57532 (4 connections now open) m31001| Fri Jan 18 13:14:10.113 [initandlisten] connection accepted from 127.0.0.4:35268 #297 (5 connections now open) m31000| Fri Jan 18 13:14:10.636 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:679482 339ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:11Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 341, "optime" : { "t" : 1358532851000, "i" : 6536 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532850000, "i" : 2448 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 306, "optime" : { "t" : 1358532851000, "i" : 4345 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 341, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 306, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 306, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 306, "optime" : { "t" : 1358532851000, "i" : 4342 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 305, "optime" : { "t" : 1358532850000, "i" : 1016 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 305, "optime" : { "t" : 1358532808000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:28Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 340, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 305, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 305, "optime" : { "t" : 1358532850000, "i" : 883 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:11Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 24 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 32 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 339, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 307, "optime" : { "t" : 1358532850000, "i" : 2448 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:11Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:11Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 306, "optime" : { "t" : 1358532851000, "i" : 5203 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 306, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 306, "optime" : { "t" : 1358532809000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:29Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 306, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 339, "optime" : { "t" : 1358532851000, "i" : 6536 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:14:13.087 [initandlisten] connection accepted from 127.0.0.5:57560 #298 (6 connections now open) m31001| Fri Jan 18 13:14:13.087 [conn294] end connection 127.0.0.5:52697 (4 connections now open) m31000| Fri Jan 18 13:14:13.367 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 locks(micros) r:145437 nreturned:160000 reslen:2720020 145ms m31000| Fri Jan 18 13:14:13.558 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 locks(micros) r:136528 nreturned:160000 reslen:2720020 136ms STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:16Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 346, "optime" : { "t" : 1358532856000, "i" : 7888 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532815000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532815000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532856000, "i" : 4871 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 311, "optime" : { "t" : 1358532855000, "i" : 2835 }, "optimeDate" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 346, "optime" : { "t" : 1358532815000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311, "optime" : { "t" : 1358532814000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 311, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311, "optime" : { "t" : 1358532855000, "i" : 2201 }, "optimeDate" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:15Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 310, "optime" : { "t" : 1358532856000, "i" : 1408 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 310, "optime" : { "t" : 1358532815000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 345, "optime" : { "t" : 1358532815000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:35Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 310, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 310, "optime" : { "t" : 1358532856000, "i" : 1308 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:16Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532814000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:15Z"), "pingMs" : 28 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532814000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 27 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 344, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 312, "optime" : { "t" : 1358532855000, "i" : 2201 }, "optimeDate" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:15Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:16Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 311, "optime" : { "t" : 1358532855000, "i" : 3300 }, "optimeDate" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311, "optime" : { "t" : 1358532814000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 311, "optime" : { "t" : 1358532814000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:34Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:16Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 311, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532856000, "i" : 7629 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:14:18.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Fri Jan 18 13:14:21.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:22Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 352, "optime" : { "t" : 1358532862000, "i" : 1147 }, "optimeDate" : ISODate("2013-01-18T18:14:22Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532819000, "i" : 4363 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532820000, "i" : 2072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532860000, "i" : 3291 }, "optimeDate" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 317, "optime" : { "t" : 1358532861000, "i" : 3338 }, "optimeDate" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 352, "optime" : { "t" : 1358532820000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 317, "optime" : { "t" : 1358532820000, "i" : 2072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 317, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 317, "optime" : { "t" : 1358532861000, "i" : 3334 }, "optimeDate" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 316, "optime" : { "t" : 1358532860000, "i" : 13 }, "optimeDate" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 316, "optime" : { "t" : 1358532819000, "i" : 4363 }, "optimeDate" : ISODate("2013-01-18T18:13:39Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 351, "optime" : { "t" : 1358532820000, "i" : 2072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 316, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 316, "optime" : { "t" : 1358532860000, "i" : 13 }, "optimeDate" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:22Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532820000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 49 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532820000, "i" : 2072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 12 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 350, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 318, "optime" : { "t" : 1358532860000, "i" : 1848 }, "optimeDate" : ISODate("2013-01-18T18:14:20Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:21Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:22Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 317, "optime" : { "t" : 1358532861000, "i" : 4095 }, "optimeDate" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 317, "optime" : { "t" : 1358532820000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 317, "optime" : { "t" : 1358532820000, "i" : 2072 }, "optimeDate" : ISODate("2013-01-18T18:13:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 317, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:21Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 350, "optime" : { "t" : 1358532862000, "i" : 911 }, "optimeDate" : ISODate("2013-01-18T18:14:22Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31003| Fri Jan 18 13:14:23.415 [conn38] end connection 127.0.0.3:59985 (3 connections now open) m31003| Fri Jan 18 13:14:23.415 [initandlisten] connection accepted from 127.0.0.3:37836 #41 (5 connections now open) m31002| Fri Jan 18 13:14:23.464 [conn279] end connection 127.0.0.6:51458 (5 connections now open) m31002| Fri Jan 18 13:14:23.465 [initandlisten] connection accepted from 127.0.0.6:39069 #283 (6 connections now open) m31003| Fri Jan 18 13:14:24.345 [conn39] end connection 127.0.0.4:42305 (3 connections now open) m31003| Fri Jan 18 13:14:24.345 [initandlisten] connection accepted from 127.0.0.4:50917 #42 (4 connections now open) m31001| Fri Jan 18 13:14:24.461 [conn295] end connection 127.0.0.2:55557 (4 connections now open) m31001| Fri Jan 18 13:14:24.461 [initandlisten] connection accepted from 127.0.0.2:47966 #299 (5 connections now open) m31000| Fri Jan 18 13:14:25.415 [conn60] end connection 127.0.0.3:38425 (11 connections now open) m31000| Fri Jan 18 13:14:25.416 [initandlisten] connection accepted from 127.0.0.3:54045 #63 (12 connections now open) m31000| Fri Jan 18 13:14:25.465 [conn61] end connection 127.0.0.6:42467 (11 connections now open) m31000| Fri Jan 18 13:14:25.465 [initandlisten] connection accepted from 127.0.0.6:54643 #64 (12 connections now open) m31000| Fri Jan 18 13:14:26.115 [conn62] end connection 127.0.0.4:43607 (11 connections now open) m31000| Fri Jan 18 13:14:26.115 [initandlisten] connection accepted from 127.0.0.4:50861 #65 (12 connections now open) STATUS FROM : 127.0.0.2:31000 m31002| Fri Jan 18 13:14:27.354 [conn280] end connection 127.0.0.5:45716 (5 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:27Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 357, "optime" : { "t" : 1358532867000, "i" : 2908 }, "optimeDate" : ISODate("2013-01-18T18:14:27Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532825000, "i" : 5336 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532825000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532866000, "i" : 4274 }, "optimeDate" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 322, "optime" : { "t" : 1358532865000, "i" : 2078 }, "optimeDate" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 357, "optime" : { "t" : 1358532825000, "i" : 5336 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 322, "optime" : { "t" : 1358532824000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 322, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 322, "optime" : { "t" : 1358532865000, "i" : 2052 }, "optimeDate" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:25Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 321, "optime" : { "t" : 1358532866000, "i" : 1273 }, "optimeDate" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 321, "optime" : { "t" : 1358532825000, "i" : 5336 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 356, "optime" : { "t" : 1358532825000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 321, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 321, "optime" : { "t" : 1358532866000, "i" : 748 }, "optimeDate" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:26Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:27Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532825000, "i" : 1905 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 40 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532824000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 13 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 355, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 323, "optime" : { "t" : 1358532865000, "i" : 2052 }, "optimeDate" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:25Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:27Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 322, "optime" : { "t" : 1358532865000, "i" : 3033 }, "optimeDate" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 322, "optime" : { "t" : 1358532825000, "i" : 5336 }, "optimeDate" : ISODate("2013-01-18T18:13:45Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 322, "optime" : { "t" : 1358532824000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:44Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:26Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 322, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 355, "optime" : { "t" : 1358532867000, "i" : 2902 }, "optimeDate" : ISODate("2013-01-18T18:14:27Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:14:27.448 [initandlisten] connection accepted from 127.0.0.5:47077 #284 (6 connections now open) m31000| Fri Jan 18 13:14:30.801 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:753848 376ms m31000| Fri Jan 18 13:14:31.144 [slaveTracking] update local.slaves query: { _id: ObjectId('50f98fbb2c4297c1fbd03d52'), host: "127.0.0.3", ns: "local.oplog.rs" } update: { $set: { syncedTo: Timestamp 1358532810000|169 } } nscanned:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:125 238ms m31000| Fri Jan 18 13:14:31.144 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:477793 238ms m31000| Fri Jan 18 13:14:31.419 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:341999 171ms m31000| Fri Jan 18 13:14:31.785 [conn22] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:221689 110ms m31003| Fri Jan 18 13:14:31.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:32Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 362, "optime" : { "t" : 1358532872000, "i" : 6157 }, "optimeDate" : ISODate("2013-01-18T18:14:32Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532829000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532829000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:49Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:32Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532870000, "i" : 69 }, "optimeDate" : ISODate("2013-01-18T18:14:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:30Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 327, "optime" : { "t" : 1358532871000, "i" : 2773 }, "optimeDate" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 362, "optime" : { "t" : 1358532831000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:51Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 327, "optime" : { "t" : 1358532830000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:32Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 327, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:31Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 327, "optime" : { "t" : 1358532871000, "i" : 2748 }, "optimeDate" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:31Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 326, "optime" : { "t" : 1358532872000, "i" : 2011 }, "optimeDate" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 326, "optime" : { "t" : 1358532831000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 361, "optime" : { "t" : 1358532831000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:51Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 326, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 326, "optime" : { "t" : 1358532872000, "i" : 1978 }, "optimeDate" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:32Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532830000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 38 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532830000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:32Z"), "pingMs" : 7 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 360, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 328, "optime" : { "t" : 1358532870000, "i" : 69 }, "optimeDate" : ISODate("2013-01-18T18:14:30Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:31Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:32Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 327, "optime" : { "t" : 1358532871000, "i" : 4083 }, "optimeDate" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 327, "optime" : { "t" : 1358532830000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 327, "optime" : { "t" : 1358532830000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:50Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:32Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 327, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:31Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 360, "optime" : { "t" : 1358532872000, "i" : 6157 }, "optimeDate" : ISODate("2013-01-18T18:14:32Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31004| Fri Jan 18 13:14:36.463 [conn46] end connection 127.0.0.2:51891 (4 connections now open) m31004| Fri Jan 18 13:14:36.463 [initandlisten] connection accepted from 127.0.0.2:39810 #49 (5 connections now open) m31004| Fri Jan 18 13:14:37.417 [conn47] end connection 127.0.0.3:58468 (4 connections now open) m31004| Fri Jan 18 13:14:37.417 [initandlisten] connection accepted from 127.0.0.3:42012 #50 (5 connections now open) m31003| Fri Jan 18 13:14:37.466 [conn40] end connection 127.0.0.6:48705 (3 connections now open) m31003| Fri Jan 18 13:14:37.466 [initandlisten] connection accepted from 127.0.0.6:38454 #43 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:37Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 367, "optime" : { "t" : 1358532877000, "i" : 5717 }, "optimeDate" : ISODate("2013-01-18T18:14:37Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532835000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532835000, "i" : 6558 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532876000, "i" : 4001 }, "optimeDate" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 332, "optime" : { "t" : 1358532877000, "i" : 3410 }, "optimeDate" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 367, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 332, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 332, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 332, "optime" : { "t" : 1358532877000, "i" : 3397 }, "optimeDate" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 331, "optime" : { "t" : 1358532876000, "i" : 1128 }, "optimeDate" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:36Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 331, "optime" : { "t" : 1358532835000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:13:55Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 366, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 331, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 331, "optime" : { "t" : 1358532876000, "i" : 1106 }, "optimeDate" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:37Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 21 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 6 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 365, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 333, "optime" : { "t" : 1358532872000, "i" : 6157 }, "optimeDate" : ISODate("2013-01-18T18:14:32Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:37Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:37Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 332, "optime" : { "t" : 1358532877000, "i" : 3479 }, "optimeDate" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 332, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 332, "optime" : { "t" : 1358532836000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:13:56Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 332, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 365, "optime" : { "t" : 1358532877000, "i" : 5717 }, "optimeDate" : ISODate("2013-01-18T18:14:37Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31004| Fri Jan 18 13:14:37.897 [conn45] end connection 127.0.0.5:33355 (4 connections now open) m31004| Fri Jan 18 13:14:38.117 [conn48] end connection 127.0.0.4:47798 (3 connections now open) m31004| Fri Jan 18 13:14:38.117 [initandlisten] connection accepted from 127.0.0.4:57945 #51 (4 connections now open) m31004| Fri Jan 18 13:14:38.397 [initandlisten] connection accepted from 127.0.0.5:33946 #52 (5 connections now open) m31002| Fri Jan 18 13:14:38.464 [conn281] end connection 127.0.0.2:57709 (5 connections now open) m31002| Fri Jan 18 13:14:38.464 [initandlisten] connection accepted from 127.0.0.2:56862 #285 (6 connections now open) m31002| Fri Jan 18 13:14:39.417 [conn282] end connection 127.0.0.3:55167 (5 connections now open) m31002| Fri Jan 18 13:14:39.417 [initandlisten] connection accepted from 127.0.0.3:49592 #286 (6 connections now open) m31001| Fri Jan 18 13:14:39.467 [conn296] end connection 127.0.0.6:55263 (4 connections now open) m31001| Fri Jan 18 13:14:39.467 [initandlisten] connection accepted from 127.0.0.6:39189 #300 (5 connections now open) m31001| Fri Jan 18 13:14:40.117 [conn297] end connection 127.0.0.4:35268 (4 connections now open) m31001| Fri Jan 18 13:14:40.117 [initandlisten] connection accepted from 127.0.0.4:44771 #301 (6 connections now open) m31000| Fri Jan 18 13:14:40.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 372, "optime" : { "t" : 1358532882000, "i" : 8536 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532882000, "i" : 3752 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 337, "optime" : { "t" : 1358532881000, "i" : 4302 }, "optimeDate" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 372, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 337, "optime" : { "t" : 1358532841000, "i" : 57 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 337, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:41Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 337, "optime" : { "t" : 1358532881000, "i" : 4239 }, "optimeDate" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:41Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:42Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 336, "optime" : { "t" : 1358532882000, "i" : 1095 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 336, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 371, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 336, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 336, "optime" : { "t" : 1358532882000, "i" : 1090 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:42Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 42 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 3 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 370, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532880000, "i" : 1603 }, "optimeDate" : ISODate("2013-01-18T18:14:40Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:41Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 338, "optime" : { "t" : 1358532881000, "i" : 5260 }, "optimeDate" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 338, "optime" : { "t" : 1358532841000, "i" : 2153 }, "optimeDate" : ISODate("2013-01-18T18:14:01Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:42Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 338, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 371, "optime" : { "t" : 1358532882000, "i" : 8536 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31001| Fri Jan 18 13:14:43.757 [conn298] end connection 127.0.0.5:57560 (4 connections now open) m31001| Fri Jan 18 13:14:43.808 [initandlisten] connection accepted from 127.0.0.5:60933 #302 (5 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 378, "optime" : { "t" : 1358532888000, "i" : 2499 }, "optimeDate" : ISODate("2013-01-18T18:14:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532845000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532845000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:05Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:48Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532886000, "i" : 4242 }, "optimeDate" : ISODate("2013-01-18T18:14:46Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 343, "optime" : { "t" : 1358532887000, "i" : 3768 }, "optimeDate" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 378, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 343, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:48Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 343, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 343, "optime" : { "t" : 1358532887000, "i" : 3679 }, "optimeDate" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 342, "optime" : { "t" : 1358532888000, "i" : 1099 }, "optimeDate" : ISODate("2013-01-18T18:14:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 342, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 377, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 342, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 342, "optime" : { "t" : 1358532888000, "i" : 1082 }, "optimeDate" : ISODate("2013-01-18T18:14:48Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:48Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 45 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 11 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 376, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 344, "optime" : { "t" : 1358532882000, "i" : 8536 }, "optimeDate" : ISODate("2013-01-18T18:14:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:45Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:47Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 343, "optime" : { "t" : 1358532887000, "i" : 4156 }, "optimeDate" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 343, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 343, "optime" : { "t" : 1358532846000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:06Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:48Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 343, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 376, "optime" : { "t" : 1358532888000, "i" : 2317 }, "optimeDate" : ISODate("2013-01-18T18:14:48Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31000| Fri Jan 18 13:14:50.415 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Fri Jan 18 13:14:53.420 [initandlisten] connection accepted from 127.0.0.3:38264 #44 (5 connections now open) m31003| Fri Jan 18 13:14:53.420 [conn41] end connection 127.0.0.3:37836 (3 connections now open) STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 383, "optime" : { "t" : 1358532893000, "i" : 4160 }, "optimeDate" : ISODate("2013-01-18T18:14:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:53Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532892000, "i" : 4279 }, "optimeDate" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 348, "optime" : { "t" : 1358532893000, "i" : 3821 }, "optimeDate" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 383, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 348, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 348, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 348, "optime" : { "t" : 1358532893000, "i" : 3465 }, "optimeDate" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 347, "optime" : { "t" : 1358532892000, "i" : 1059 }, "optimeDate" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 347, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:53Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 382, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 347, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 347, "optime" : { "t" : 1358532892000, "i" : 78 }, "optimeDate" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:53Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532851000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:11Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:53Z"), "pingMs" : 35 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532850000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:52Z"), "pingMs" : 8 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 381, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 349, "optime" : { "t" : 1358532893000, "i" : 3465 }, "optimeDate" : ISODate("2013-01-18T18:14:53Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 348, "optime" : { "t" : 1358532891000, "i" : 1670 }, "optimeDate" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 348, "optime" : { "t" : 1358532850000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:53Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 348, "optime" : { "t" : 1358532850000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:10Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:52Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 348, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 381, "optime" : { "t" : 1358532893000, "i" : 4160 }, "optimeDate" : ISODate("2013-01-18T18:14:53Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } m31002| Fri Jan 18 13:14:53.469 [conn283] end connection 127.0.0.6:39069 (5 connections now open) m31002| Fri Jan 18 13:14:53.469 [initandlisten] connection accepted from 127.0.0.6:54414 #287 (6 connections now open) m31003| Fri Jan 18 13:14:53.859 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Fri Jan 18 13:14:54.350 [conn42] end connection 127.0.0.4:50917 (3 connections now open) m31003| Fri Jan 18 13:14:54.350 [initandlisten] connection accepted from 127.0.0.4:45487 #45 (4 connections now open) m31001| Fri Jan 18 13:14:54.466 [conn299] end connection 127.0.0.2:47966 (4 connections now open) m31001| Fri Jan 18 13:14:54.466 [initandlisten] connection accepted from 127.0.0.2:35209 #303 (5 connections now open) m31000| Fri Jan 18 13:14:55.193 [conn32] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313507866586101 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:184065 nreturned:160000 reslen:2720020 143ms m31000| Fri Jan 18 13:14:55.198 [conn33] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1358532544000|1 } } cursorid:313508297603461 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:188141 nreturned:160000 reslen:2720020 148ms m31000| Fri Jan 18 13:14:55.420 [conn63] end connection 127.0.0.3:54045 (11 connections now open) m31000| Fri Jan 18 13:14:55.420 [initandlisten] connection accepted from 127.0.0.3:50202 #66 (12 connections now open) m31000| Fri Jan 18 13:14:55.469 [conn64] end connection 127.0.0.6:54643 (11 connections now open) m31000| Fri Jan 18 13:14:55.469 [initandlisten] connection accepted from 127.0.0.6:60334 #67 (12 connections now open) m31000| Fri Jan 18 13:14:56.120 [conn65] end connection 127.0.0.4:50861 (11 connections now open) m31000| Fri Jan 18 13:14:56.120 [initandlisten] connection accepted from 127.0.0.4:56988 #68 (12 connections now open) m31002| Fri Jan 18 13:14:57.585 [conn284] end connection 127.0.0.5:47077 (5 connections now open) m31002| Fri Jan 18 13:14:57.679 [initandlisten] connection accepted from 127.0.0.5:59095 #288 (6 connections now open) m31003| Fri Jan 18 13:14:58.400 [rsMgr] replSet I don't see a primary and I can't elect myself STATUS FROM : 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 388, "optime" : { "t" : 1358532898000, "i" : 6360 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532515000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:08:35Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532898000, "i" : 3972 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 353, "optime" : { "t" : 1358532897000, "i" : 3648 }, "optimeDate" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 388, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 353, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 353, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 353, "optime" : { "t" : 1358532897000, "i" : 3648 }, "optimeDate" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:57Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.4:31002 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 352, "optime" : { "t" : 1358532898000, "i" : 147 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 352, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 387, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 352, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 352, "optime" : { "t" : 1358532898000, "i" : 135 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } STATUS FROM : 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:58Z"), "myState" : 3, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1358532582000, "i" : 1 }, "optimeDate" : ISODate("2013-01-18T18:09:42Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 17 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 6 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 386, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "errmsg" : "still syncing, not yet to minValid optime 50f99053:206f", "self" : true }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 354, "optime" : { "t" : 1358532898000, "i" : 3790 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 500 } ], "ok" : 1 } STATUS FROM : 127.0.0.6:31004 { "set" : "testReplSet", "date" : ISODate("2013-01-18T18:14:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 353, "optime" : { "t" : 1358532897000, "i" : 4355 }, "optimeDate" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 353, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 353, "optime" : { "t" : 1358532856000, "i" : 5002 }, "optimeDate" : ISODate("2013-01-18T18:14:16Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0 }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 353, "optime" : { "t" : 1358532599000, "i" : 355 }, "optimeDate" : ISODate("2013-01-18T18:09:59Z"), "lastHeartbeat" : ISODate("2013-01-18T18:14:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-18T18:14:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50f99053:206f" }, { "_id" : 4, "name" : "127.0.0.6:31004", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 386, "optime" : { "t" : 1358532898000, "i" : 6360 }, "optimeDate" : ISODate("2013-01-18T18:14:58Z"), "self" : true } ], "ok" : 1 } { "roots" : [ "127.0.0.2:31000" ], "graph" : { "127.0.0.2:31000" : [ "127.0.0.3:31001", "127.0.0.4:31002", "127.0.0.6:31004" ], "127.0.0.4:31002" : [ "127.0.0.5:31003" ] } } Fri Jan 18 13:15:03.765 got signal 2 (Interrupt), will terminate after current cmd ends Fri Jan 18 13:15:03.766 [interruptThread] now exiting Fri Jan 18 13:15:03.766 dbexit: Fri Jan 18 13:15:03.766 [interruptThread] shutdown: going to close listening sockets... Fri Jan 18 13:15:03.766 [interruptThread] closing listening socket: 9 Fri Jan 18 13:15:03.766 [interruptThread] closing listening socket: 10 Fri Jan 18 13:15:03.766 [interruptThread] closing listening socket: 11 Fri Jan 18 13:15:03.766 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Fri Jan 18 13:15:03.766 [interruptThread] shutdown: going to flush diaglog... Fri Jan 18 13:15:03.766 [interruptThread] shutdown: going to close sockets... Fri Jan 18 13:15:03.766 [interruptThread] shutdown: waiting for fs preallocator... Fri Jan 18 13:15:03.766 [interruptThread] shutdown: lock for final commit... Fri Jan 18 13:15:03.766 [interruptThread] shutdown: final commit... Fri Jan 18 13:15:03.773 [interruptThread] shutdown: closing all files... Fri Jan 18 13:15:03.773 [interruptThread] closeAllFiles() finished Fri Jan 18 13:15:03.773 [interruptThread] journalCleanup... Fri Jan 18 13:15:03.773 [interruptThread] removeJournalFiles Fri Jan 18 13:15:04.289 [interruptThread] shutdown: removing fs lock... Fri Jan 18 13:15:04.304 dbexit: really exiting now 0 tests succeeded 1 tests didn't get run