cwd [/home/gregorv/Workspaces/10Gen/mongo] num procs:222 removing: /data/db/sconsTests//local.ns removing: /data/db/sconsTests//mongod.lock removing: /data/db/sconsTests//local.0 Thu Jan 17 16:56:04.689 [initandlisten] MongoDB starting : pid=30968 port=27999 dbpath=/data/db/sconsTests/ 64-bit host=nuwen Thu Jan 17 16:56:04.689 [initandlisten] Thu Jan 17 16:56:04.689 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. Thu Jan 17 16:56:04.689 [initandlisten] ** Not recommended for production. Thu Jan 17 16:56:04.689 [initandlisten] Thu Jan 17 16:56:04.689 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 Thu Jan 17 16:56:04.689 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 Thu Jan 17 16:56:04.689 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 Thu Jan 17 16:56:04.689 [initandlisten] allocator: tcmalloc Thu Jan 17 16:56:04.689 [initandlisten] options: { dbpath: "/data/db/sconsTests/", nopreallocj: true, port: 27999, setParameter: [ "enableTestCommands=1" ] } Thu Jan 17 16:56:04.747 [initandlisten] journal dir=/data/db/sconsTests/journal Thu Jan 17 16:56:04.747 [initandlisten] recover : no journal files present, no recovery needed Thu Jan 17 16:56:04.798 [FileAllocator] allocating new datafile /data/db/sconsTests/local.ns, filling with zeroes... Thu Jan 17 16:56:04.798 [FileAllocator] creating directory /data/db/sconsTests/_tmp Thu Jan 17 16:56:04.990 [FileAllocator] done allocating datafile /data/db/sconsTests/local.ns, size: 16MB, took 0.15 secs Thu Jan 17 16:56:04.990 [FileAllocator] allocating new datafile /data/db/sconsTests/local.0, filling with zeroes... Thu Jan 17 16:56:05.514 [FileAllocator] done allocating datafile /data/db/sconsTests/local.0, size: 64MB, took 0.524 secs Thu Jan 17 16:56:05.516 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 717ms Thu Jan 17 16:56:05.516 [initandlisten] waiting for connections on port 27999 Thu Jan 17 16:56:05.516 [websvr] admin web console waiting for connections on port 28999 running /home/gregorv/Workspaces/10Gen/mongo/mongod --port 27999 --dbpath /data/db/sconsTests/ --setParameter enableTestCommands=1 --nopreallocj ******************************************* Test : sync_change_source.js ... Command : /home/gregorv/Workspaces/10Gen/mongo/mongo --port 27999 --authenticationMechanism MONGO-CR /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js --eval TestData = new Object();TestData.testPath = "/home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js";TestData.testFile = "sync_change_source.js";TestData.testName = "sync_change_source";TestData.noJournal = false;TestData.noJournalPrealloc = true;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null; Date : Thu Jan 17 16:56:05 2013 MongoDB shell version: 2.3.3-pre- connecting to: 127.0.0.1:27999/test Thu Jan 17 16:56:05.694 [initandlisten] connection accepted from 127.0.0.1:56757 #2 (1 connection now open) null Running: sudo bash -c tc qdisc del root dev lo > /tmp/output Thu Jan 17 16:56:05.699 shell: started program sudo bash -c tc qdisc del root dev lo > /tmp/output Running: sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Thu Jan 17 16:56:05.712 shell: started program sudo bash -c tc qdisc add dev lo handle 1: root htb > /tmp/output Running: sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Thu Jan 17 16:56:05.726 shell: started program sudo bash -c tc class add dev lo parent 1: classid 1:1 htb rate 1000Mbps > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output Thu Jan 17 16:56:05.740 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:10 htb rate 1bps ceil 1000Mbps > /tmp/output ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002, 31003 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.2 Resetting db path '/data/db/testReplSet-0' Thu Jan 17 16:56:05.798 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-0 --nopreallocj --bind_ip 127.0.0.2 --setParameter enableTestCommands=1 m31000| note: noprealloc may hurt performance in many applications m31000| Thu Jan 17 16:56:05.826 [initandlisten] MongoDB starting : pid=31010 port=31000 dbpath=/data/db/testReplSet-0 64-bit host=nuwen m31000| Thu Jan 17 16:56:05.826 [initandlisten] m31000| Thu Jan 17 16:56:05.826 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31000| Thu Jan 17 16:56:05.826 [initandlisten] ** Not recommended for production. m31000| Thu Jan 17 16:56:05.826 [initandlisten] m31000| Thu Jan 17 16:56:05.826 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31000| Thu Jan 17 16:56:05.826 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31000| Thu Jan 17 16:56:05.826 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31000| Thu Jan 17 16:56:05.826 [initandlisten] allocator: tcmalloc m31000| Thu Jan 17 16:56:05.826 [initandlisten] options: { bind_ip: "127.0.0.2", dbpath: "/data/db/testReplSet-0", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31000, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31000| Thu Jan 17 16:56:05.900 [initandlisten] journal dir=/data/db/testReplSet-0/journal m31000| Thu Jan 17 16:56:05.900 [initandlisten] recover : no journal files present, no recovery needed m31000| Thu Jan 17 16:56:05.942 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.ns, filling with zeroes... m31000| Thu Jan 17 16:56:05.942 [FileAllocator] creating directory /data/db/testReplSet-0/_tmp m31000| Thu Jan 17 16:56:06.125 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.ns, size: 16MB, took 0.149 secs m31000| Thu Jan 17 16:56:06.126 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.0, filling with zeroes... m31000| Thu Jan 17 16:56:06.275 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.0, size: 16MB, took 0.149 secs m31000| Thu Jan 17 16:56:06.277 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 334ms m31000| Thu Jan 17 16:56:06.277 [initandlisten] waiting for connections on port 31000 m31000| Thu Jan 17 16:56:06.277 [websvr] admin web console waiting for connections on port 32000 m31000| Thu Jan 17 16:56:06.279 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Thu Jan 17 16:56:06.279 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Thu Jan 17 16:56:06.400 [initandlisten] connection accepted from 127.0.0.1:56467 #1 (1 connection now open) [ connection to 127.0.0.2:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002, 31003 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.3 Resetting db path '/data/db/testReplSet-1' Thu Jan 17 16:56:06.437 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-1 --nopreallocj --bind_ip 127.0.0.3 --setParameter enableTestCommands=1 m31001| note: noprealloc may hurt performance in many applications m31001| Thu Jan 17 16:56:06.464 [initandlisten] MongoDB starting : pid=31062 port=31001 dbpath=/data/db/testReplSet-1 64-bit host=nuwen m31001| Thu Jan 17 16:56:06.464 [initandlisten] m31001| Thu Jan 17 16:56:06.464 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31001| Thu Jan 17 16:56:06.464 [initandlisten] ** Not recommended for production. m31001| Thu Jan 17 16:56:06.464 [initandlisten] m31001| Thu Jan 17 16:56:06.464 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31001| Thu Jan 17 16:56:06.464 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31001| Thu Jan 17 16:56:06.464 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31001| Thu Jan 17 16:56:06.464 [initandlisten] allocator: tcmalloc m31001| Thu Jan 17 16:56:06.464 [initandlisten] options: { bind_ip: "127.0.0.3", dbpath: "/data/db/testReplSet-1", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31001, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31001| Thu Jan 17 16:56:06.518 [initandlisten] journal dir=/data/db/testReplSet-1/journal m31001| Thu Jan 17 16:56:06.518 [initandlisten] recover : no journal files present, no recovery needed m31001| Thu Jan 17 16:56:06.569 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.ns, filling with zeroes... m31001| Thu Jan 17 16:56:06.569 [FileAllocator] creating directory /data/db/testReplSet-1/_tmp m31001| Thu Jan 17 16:56:06.743 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.ns, size: 16MB, took 0.141 secs m31001| Thu Jan 17 16:56:06.744 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.0, filling with zeroes... m31001| Thu Jan 17 16:56:06.894 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.0, size: 16MB, took 0.149 secs m31001| Thu Jan 17 16:56:06.895 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 325ms m31001| Thu Jan 17 16:56:06.895 [initandlisten] waiting for connections on port 31001 m31001| Thu Jan 17 16:56:06.895 [websvr] admin web console waiting for connections on port 32001 m31001| Thu Jan 17 16:56:06.897 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Thu Jan 17 16:56:06.897 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Thu Jan 17 16:56:07.039 [initandlisten] connection accepted from 127.0.0.1:59587 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002, 31003 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.4 Resetting db path '/data/db/testReplSet-2' Thu Jan 17 16:56:07.074 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31002 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-2 --nopreallocj --bind_ip 127.0.0.4 --setParameter enableTestCommands=1 m31002| note: noprealloc may hurt performance in many applications m31002| Thu Jan 17 16:56:07.098 [initandlisten] MongoDB starting : pid=31113 port=31002 dbpath=/data/db/testReplSet-2 64-bit host=nuwen m31002| Thu Jan 17 16:56:07.098 [initandlisten] m31002| Thu Jan 17 16:56:07.098 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31002| Thu Jan 17 16:56:07.099 [initandlisten] ** Not recommended for production. m31002| Thu Jan 17 16:56:07.099 [initandlisten] m31002| Thu Jan 17 16:56:07.099 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31002| Thu Jan 17 16:56:07.099 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31002| Thu Jan 17 16:56:07.099 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31002| Thu Jan 17 16:56:07.099 [initandlisten] allocator: tcmalloc m31002| Thu Jan 17 16:56:07.099 [initandlisten] options: { bind_ip: "127.0.0.4", dbpath: "/data/db/testReplSet-2", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31002, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31002| Thu Jan 17 16:56:07.161 [initandlisten] journal dir=/data/db/testReplSet-2/journal m31002| Thu Jan 17 16:56:07.161 [initandlisten] recover : no journal files present, no recovery needed m31002| Thu Jan 17 16:56:07.212 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.ns, filling with zeroes... m31002| Thu Jan 17 16:56:07.212 [FileAllocator] creating directory /data/db/testReplSet-2/_tmp m31002| Thu Jan 17 16:56:07.395 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.ns, size: 16MB, took 0.149 secs m31002| Thu Jan 17 16:56:07.395 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.0, filling with zeroes... m31002| Thu Jan 17 16:56:07.545 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.0, size: 16MB, took 0.149 secs m31002| Thu Jan 17 16:56:07.546 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 334ms m31002| Thu Jan 17 16:56:07.546 [initandlisten] waiting for connections on port 31002 m31002| Thu Jan 17 16:56:07.547 [websvr] admin web console waiting for connections on port 32002 m31002| Thu Jan 17 16:56:07.549 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Thu Jan 17 16:56:07.549 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Thu Jan 17 16:56:07.676 [initandlisten] connection accepted from 127.0.0.1:60487 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002 ] ReplSetTest n is : 3 ReplSetTest n: 3 ports: [ 31000, 31001, 31002, 31003 ] 31003 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31003, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testReplSet", "dbpath" : "$set-$node", "useIP" : true, "restart" : undefined, "pathOpts" : { "node" : 3, "set" : "testReplSet" } } ReplSetTest Starting.... Using Local IP for host: 127.0.0.5 Resetting db path '/data/db/testReplSet-3' Thu Jan 17 16:56:07.712 shell: started program /home/gregorv/Workspaces/10Gen/mongo/mongod --oplogSize 40 --port 31003 --noprealloc --smallfiles --rest --replSet testReplSet --dbpath /data/db/testReplSet-3 --nopreallocj --bind_ip 127.0.0.5 --setParameter enableTestCommands=1 m31003| note: noprealloc may hurt performance in many applications m31003| Thu Jan 17 16:56:07.737 [initandlisten] MongoDB starting : pid=31165 port=31003 dbpath=/data/db/testReplSet-3 64-bit host=nuwen m31003| Thu Jan 17 16:56:07.738 [initandlisten] m31003| Thu Jan 17 16:56:07.738 [initandlisten] ** NOTE: This is a development version (2.3.3-pre-) of MongoDB. m31003| Thu Jan 17 16:56:07.738 [initandlisten] ** Not recommended for production. m31003| Thu Jan 17 16:56:07.738 [initandlisten] m31003| Thu Jan 17 16:56:07.738 [initandlisten] db version v2.3.3-pre-, pdfile version 4.5 m31003| Thu Jan 17 16:56:07.738 [initandlisten] git version: b8c0b6ae71fb6c076101fa25ef915021ea26e156 m31003| Thu Jan 17 16:56:07.738 [initandlisten] build info: Linux nuwen 3.5.0-21-generic #32-Ubuntu SMP Tue Dec 11 18:51:59 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31003| Thu Jan 17 16:56:07.738 [initandlisten] allocator: tcmalloc m31003| Thu Jan 17 16:56:07.738 [initandlisten] options: { bind_ip: "127.0.0.5", dbpath: "/data/db/testReplSet-3", noprealloc: true, nopreallocj: true, oplogSize: 40, port: 31003, replSet: "testReplSet", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31003| Thu Jan 17 16:56:07.804 [initandlisten] journal dir=/data/db/testReplSet-3/journal m31003| Thu Jan 17 16:56:07.804 [initandlisten] recover : no journal files present, no recovery needed m31003| Thu Jan 17 16:56:07.847 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.ns, filling with zeroes... m31003| Thu Jan 17 16:56:07.847 [FileAllocator] creating directory /data/db/testReplSet-3/_tmp m31003| Thu Jan 17 16:56:08.030 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.ns, size: 16MB, took 0.149 secs m31003| Thu Jan 17 16:56:08.030 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.0, filling with zeroes... m31003| Thu Jan 17 16:56:08.171 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.0, size: 16MB, took 0.141 secs m31003| Thu Jan 17 16:56:08.172 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 325ms m31003| Thu Jan 17 16:56:08.173 [initandlisten] waiting for connections on port 31003 m31003| Thu Jan 17 16:56:08.173 [websvr] admin web console waiting for connections on port 32003 m31003| Thu Jan 17 16:56:08.174 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31003| Thu Jan 17 16:56:08.174 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31003| Thu Jan 17 16:56:08.313 [initandlisten] connection accepted from 127.0.0.1:49675 #1 (1 connection now open) [ connection to 127.0.0.2:31000, connection to 127.0.0.3:31001, connection to 127.0.0.4:31002, connection to 127.0.0.5:31003 ] { "replSetInitiate" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000" }, { "_id" : 1, "host" : "127.0.0.3:31001" }, { "_id" : 2, "host" : "127.0.0.4:31002" }, { "_id" : 3, "host" : "127.0.0.5:31003" } ] } } m31000| Thu Jan 17 16:56:08.316 [conn1] replSet replSetInitiate admin command received from client m31000| Thu Jan 17 16:56:08.316 [conn1] replSet replSetInitiate config object parses ok, 4 members specified m31001| Thu Jan 17 16:56:08.317 [initandlisten] connection accepted from 127.0.0.2:39758 #2 (2 connections now open) m31002| Thu Jan 17 16:56:08.318 [initandlisten] connection accepted from 127.0.0.2:45916 #2 (2 connections now open) m31003| Thu Jan 17 16:56:08.318 [initandlisten] connection accepted from 127.0.0.2:60206 #2 (2 connections now open) m31000| Thu Jan 17 16:56:08.319 [conn1] replSet replSetInitiate all members seem up m31000| Thu Jan 17 16:56:08.319 [conn1] ****** m31000| Thu Jan 17 16:56:08.319 [conn1] creating replication oplog of size: 40MB... m31000| Thu Jan 17 16:56:08.319 [FileAllocator] allocating new datafile /data/db/testReplSet-0/local.1, filling with zeroes... m31000| Thu Jan 17 16:56:08.846 [FileAllocator] done allocating datafile /data/db/testReplSet-0/local.1, size: 64MB, took 0.526 secs m31000| Thu Jan 17 16:56:09.897 [conn1] ****** m31000| Thu Jan 17 16:56:09.897 [conn1] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:56:09.898 [conn1] replSet saveConfigLocally done m31000| Thu Jan 17 16:56:09.898 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Thu Jan 17 16:56:09.898 [conn1] build index local.replset.minvalid { _id: 1 } m31000| Thu Jan 17 16:56:09.899 [conn1] build index done. scanned 0 total records. 0 secs m31000| Thu Jan 17 16:56:09.899 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000" }, { _id: 1.0, host: "127.0.0.3:31001" }, { _id: 2.0, host: "127.0.0.4:31002" }, { _id: 3.0, host: "127.0.0.5:31003" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1580135 reslen:112 1583ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Thu Jan 17 16:56:16.280 [rsStart] replSet I am 127.0.0.2:31000 m31000| Thu Jan 17 16:56:16.280 [rsStart] replSet STARTUP2 m31000| Thu Jan 17 16:56:16.280 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:56:16.280 [conn2] end connection 127.0.0.2:39758 (1 connection now open) m31000| Thu Jan 17 16:56:16.281 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31000| Thu Jan 17 16:56:16.281 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:56:16.281 [initandlisten] connection accepted from 127.0.0.2:59723 #3 (2 connections now open) m31000| Thu Jan 17 16:56:16.281 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31001| Thu Jan 17 16:56:16.898 [rsStart] trying to contact 127.0.0.2:31000 m31000| Thu Jan 17 16:56:16.898 [initandlisten] connection accepted from 127.0.0.3:46504 #2 (2 connections now open) m31001| Thu Jan 17 16:56:16.899 [rsStart] replSet I am 127.0.0.3:31001 m31001| Thu Jan 17 16:56:16.899 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Thu Jan 17 16:56:16.899 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:56:17.117 [rsStart] replSet saveConfigLocally done m31001| Thu Jan 17 16:56:17.118 [rsStart] replSet STARTUP2 m31001| Thu Jan 17 16:56:17.118 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31001| Thu Jan 17 16:56:17.118 [rsSync] ****** m31001| Thu Jan 17 16:56:17.118 [rsSync] creating replication oplog of size: 40MB... m31001| Thu Jan 17 16:56:17.160 [FileAllocator] allocating new datafile /data/db/testReplSet-1/local.1, filling with zeroes... m31000| Thu Jan 17 16:56:17.281 [rsSync] replSet SECONDARY m31002| Thu Jan 17 16:56:17.549 [rsStart] trying to contact 127.0.0.2:31000 m31000| Thu Jan 17 16:56:17.549 [initandlisten] connection accepted from 127.0.0.4:44584 #3 (3 connections now open) m31002| Thu Jan 17 16:56:17.550 [rsStart] replSet I am 127.0.0.4:31002 m31002| Thu Jan 17 16:56:17.550 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Thu Jan 17 16:56:17.550 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:56:17.783 [rsStart] replSet saveConfigLocally done m31001| Thu Jan 17 16:56:17.783 [FileAllocator] done allocating datafile /data/db/testReplSet-1/local.1, size: 64MB, took 0.623 secs m31002| Thu Jan 17 16:56:17.784 [rsStart] replSet STARTUP2 m31002| Thu Jan 17 16:56:17.784 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31002| Thu Jan 17 16:56:17.784 [rsSync] ****** m31002| Thu Jan 17 16:56:17.784 [rsSync] creating replication oplog of size: 40MB... m31002| Thu Jan 17 16:56:18.001 [FileAllocator] allocating new datafile /data/db/testReplSet-2/local.1, filling with zeroes... m31003| Thu Jan 17 16:56:18.175 [rsStart] trying to contact 127.0.0.2:31000 m31000| Thu Jan 17 16:56:18.175 [initandlisten] connection accepted from 127.0.0.5:56449 #4 (4 connections now open) m31003| Thu Jan 17 16:56:18.175 [rsStart] replSet I am 127.0.0.5:31003 m31003| Thu Jan 17 16:56:18.175 [rsStart] replSet got config version 1 from a remote, saving locally m31003| Thu Jan 17 16:56:18.175 [rsStart] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:56:18.280 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31000| Thu Jan 17 16:56:18.280 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31000| Thu Jan 17 16:56:18.281 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31000| Thu Jan 17 16:56:18.282 [rsHealthPoll] replset info 127.0.0.3:31001 thinks that we are down m31000| Thu Jan 17 16:56:18.282 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31000| Thu Jan 17 16:56:18.282 [rsMgr] not electing self, 127.0.0.4:31002 would veto with 'I don't think 127.0.0.2:31000 is electable' m31002| Thu Jan 17 16:56:18.825 [FileAllocator] done allocating datafile /data/db/testReplSet-2/local.1, size: 64MB, took 0.824 secs m31001| Thu Jan 17 16:56:18.825 [rsSync] ****** m31001| Thu Jan 17 16:56:18.825 [rsSync] replSet initial sync pending m31001| Thu Jan 17 16:56:18.825 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31003| Thu Jan 17 16:56:18.840 [rsStart] replSet saveConfigLocally done m31003| Thu Jan 17 16:56:18.840 [rsStart] replSet STARTUP2 m31003| Thu Jan 17 16:56:18.841 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31003| Thu Jan 17 16:56:18.841 [rsSync] ****** m31003| Thu Jan 17 16:56:18.841 [rsSync] creating replication oplog of size: 40MB... m31002| Thu Jan 17 16:56:18.899 [initandlisten] connection accepted from 127.0.0.3:48787 #3 (3 connections now open) m31000| Thu Jan 17 16:56:18.899 [conn2] end connection 127.0.0.3:46504 (3 connections now open) m31003| Thu Jan 17 16:56:18.899 [initandlisten] connection accepted from 127.0.0.3:42339 #3 (3 connections now open) m31000| Thu Jan 17 16:56:18.899 [initandlisten] connection accepted from 127.0.0.3:49790 #5 (5 connections now open) m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replset info 127.0.0.4:31002 thinks that we are down m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replset info 127.0.0.5:31003 thinks that we are down m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:56:18.900 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Thu Jan 17 16:56:19.076 [FileAllocator] allocating new datafile /data/db/testReplSet-3/local.1, filling with zeroes... m31000| Thu Jan 17 16:56:19.550 [conn3] end connection 127.0.0.4:44584 (3 connections now open) m31001| Thu Jan 17 16:56:19.550 [initandlisten] connection accepted from 127.0.0.4:52793 #4 (3 connections now open) m31003| Thu Jan 17 16:56:19.550 [initandlisten] connection accepted from 127.0.0.4:54529 #4 (4 connections now open) m31000| Thu Jan 17 16:56:19.550 [initandlisten] connection accepted from 127.0.0.4:57313 #6 (4 connections now open) m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replset info 127.0.0.5:31003 thinks that we are down m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:56:19.551 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31002| Thu Jan 17 16:56:19.742 [rsSync] ****** m31003| Thu Jan 17 16:56:19.742 [FileAllocator] done allocating datafile /data/db/testReplSet-3/local.1, size: 64MB, took 0.665 secs m31002| Thu Jan 17 16:56:19.742 [rsSync] replSet initial sync pending m31002| Thu Jan 17 16:56:19.742 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Thu Jan 17 16:56:20.176 [conn4] end connection 127.0.0.5:56449 (3 connections now open) m31001| Thu Jan 17 16:56:20.176 [initandlisten] connection accepted from 127.0.0.5:44504 #5 (4 connections now open) m31000| Thu Jan 17 16:56:20.176 [initandlisten] connection accepted from 127.0.0.5:43645 #7 (4 connections now open) m31002| Thu Jan 17 16:56:20.176 [initandlisten] connection accepted from 127.0.0.5:56462 #4 (4 connections now open) m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state STARTUP2 m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:56:20.176 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state STARTUP2 m31000| Thu Jan 17 16:56:20.281 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state STARTUP2 m31000| Thu Jan 17 16:56:20.282 [rsMgr] not electing self, 127.0.0.5:31003 would veto with 'I don't think 127.0.0.2:31000 is electable' m31003| Thu Jan 17 16:56:20.609 [rsSync] ****** m31003| Thu Jan 17 16:56:20.609 [rsSync] replSet initial sync pending m31003| Thu Jan 17 16:56:20.609 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Thu Jan 17 16:56:26.282 [rsMgr] replSet info electSelf 0 m31001| Thu Jan 17 16:56:26.282 [conn3] replSet RECOVERING m31001| Thu Jan 17 16:56:26.282 [conn3] replSet info voting yea for 127.0.0.2:31000 (0) m31003| Thu Jan 17 16:56:26.282 [conn2] replSet RECOVERING m31003| Thu Jan 17 16:56:26.282 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31002| Thu Jan 17 16:56:26.282 [conn2] replSet RECOVERING m31002| Thu Jan 17 16:56:26.283 [conn2] replSet info voting yea for 127.0.0.2:31000 (0) m31000| Thu Jan 17 16:56:26.283 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31000| Thu Jan 17 16:56:26.283 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31001| Thu Jan 17 16:56:26.901 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31001| Thu Jan 17 16:56:26.901 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31000| Thu Jan 17 16:56:27.282 [rsMgr] replSet PRIMARY m31002| Thu Jan 17 16:56:27.552 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:56:27.552 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state RECOVERING m31002| Thu Jan 17 16:56:27.552 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31003| Thu Jan 17 16:56:28.178 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31003| Thu Jan 17 16:56:28.178 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state RECOVERING m31003| Thu Jan 17 16:56:28.178 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31000| Thu Jan 17 16:56:28.282 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state RECOVERING m31001| Thu Jan 17 16:56:28.901 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:56:30.282 [conn2] end connection 127.0.0.2:45916 (3 connections now open) m31002| Thu Jan 17 16:56:30.282 [initandlisten] connection accepted from 127.0.0.2:47048 #5 (4 connections now open) m31002| Thu Jan 17 16:56:32.902 [conn3] end connection 127.0.0.3:48787 (3 connections now open) m31002| Thu Jan 17 16:56:32.902 [initandlisten] connection accepted from 127.0.0.3:45160 #6 (4 connections now open) m31001| Thu Jan 17 16:56:33.553 [conn4] end connection 127.0.0.4:52793 (3 connections now open) m31001| Thu Jan 17 16:56:33.553 [initandlisten] connection accepted from 127.0.0.4:39510 #6 (5 connections now open) m31001| Thu Jan 17 16:56:34.179 [conn5] end connection 127.0.0.5:44504 (3 connections now open) m31001| Thu Jan 17 16:56:34.179 [initandlisten] connection accepted from 127.0.0.5:46858 #7 (4 connections now open) m31001| Thu Jan 17 16:56:34.825 [rsSync] replSet initial sync pending m31001| Thu Jan 17 16:56:34.825 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:34.826 [initandlisten] connection accepted from 127.0.0.3:60586 #8 (5 connections now open) m31001| Thu Jan 17 16:56:34.827 [rsSync] build index local.me { _id: 1 } m31001| Thu Jan 17 16:56:34.827 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Thu Jan 17 16:56:34.828 [rsSync] replSet initial sync drop all databases m31001| Thu Jan 17 16:56:34.828 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Thu Jan 17 16:56:34.828 [rsSync] replSet initial sync clone all databases m31001| Thu Jan 17 16:56:34.829 [rsSync] replSet initial sync data copy, starting syncup m31001| Thu Jan 17 16:56:34.829 [rsSync] oplog sync 1 of 3 m31001| Thu Jan 17 16:56:34.829 [rsSync] oplog sync 2 of 3 m31001| Thu Jan 17 16:56:34.829 [rsSync] replSet initial sync building indexes m31001| Thu Jan 17 16:56:34.829 [rsSync] oplog sync 3 of 3 m31001| Thu Jan 17 16:56:34.829 [rsSync] replSet initial sync finishing up m31001| Thu Jan 17 16:56:34.830 [rsSync] replSet set minValid=50f87379:1 m31001| Thu Jan 17 16:56:34.830 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Thu Jan 17 16:56:34.830 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Thu Jan 17 16:56:34.831 [rsSync] replSet initial sync done m31000| Thu Jan 17 16:56:34.831 [conn8] end connection 127.0.0.3:60586 (4 connections now open) m31001| Thu Jan 17 16:56:35.120 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:35.120 [initandlisten] connection accepted from 127.0.0.3:33665 #9 (5 connections now open) m31002| Thu Jan 17 16:56:35.742 [rsSync] replSet initial sync pending m31002| Thu Jan 17 16:56:35.742 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:35.743 [initandlisten] connection accepted from 127.0.0.4:36730 #10 (6 connections now open) m31002| Thu Jan 17 16:56:35.744 [rsSync] build index local.me { _id: 1 } m31002| Thu Jan 17 16:56:35.744 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Thu Jan 17 16:56:35.745 [rsSync] replSet initial sync drop all databases m31002| Thu Jan 17 16:56:35.745 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Thu Jan 17 16:56:35.745 [rsSync] replSet initial sync clone all databases m31002| Thu Jan 17 16:56:35.745 [rsSync] replSet initial sync data copy, starting syncup m31002| Thu Jan 17 16:56:35.746 [rsSync] oplog sync 1 of 3 m31002| Thu Jan 17 16:56:35.746 [rsSync] oplog sync 2 of 3 m31002| Thu Jan 17 16:56:35.746 [rsSync] replSet initial sync building indexes m31002| Thu Jan 17 16:56:35.746 [rsSync] oplog sync 3 of 3 m31002| Thu Jan 17 16:56:35.746 [rsSync] replSet initial sync finishing up m31002| Thu Jan 17 16:56:35.747 [rsSync] replSet set minValid=50f87379:1 m31002| Thu Jan 17 16:56:35.747 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Thu Jan 17 16:56:35.748 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Thu Jan 17 16:56:35.748 [rsSync] replSet initial sync done m31000| Thu Jan 17 16:56:35.748 [conn10] end connection 127.0.0.4:36730 (5 connections now open) m31002| Thu Jan 17 16:56:35.785 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:35.786 [initandlisten] connection accepted from 127.0.0.4:41208 #11 (6 connections now open) m31001| Thu Jan 17 16:56:35.831 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:56:35.831 [initandlisten] connection accepted from 127.0.0.3:51989 #12 (7 connections now open) m31003| Thu Jan 17 16:56:36.609 [rsSync] replSet initial sync pending m31003| Thu Jan 17 16:56:36.609 [rsSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:36.609 [initandlisten] connection accepted from 127.0.0.5:48716 #13 (8 connections now open) m31003| Thu Jan 17 16:56:36.610 [rsSync] build index local.me { _id: 1 } m31003| Thu Jan 17 16:56:36.610 [rsSync] build index done. scanned 0 total records. 0 secs m31003| Thu Jan 17 16:56:36.611 [rsSync] replSet initial sync drop all databases m31003| Thu Jan 17 16:56:36.611 [rsSync] dropAllDatabasesExceptLocal 1 m31003| Thu Jan 17 16:56:36.611 [rsSync] replSet initial sync clone all databases m31003| Thu Jan 17 16:56:36.611 [rsSync] replSet initial sync data copy, starting syncup m31003| Thu Jan 17 16:56:36.611 [rsSync] oplog sync 1 of 3 m31003| Thu Jan 17 16:56:36.611 [rsSync] oplog sync 2 of 3 m31003| Thu Jan 17 16:56:36.611 [rsSync] replSet initial sync building indexes m31003| Thu Jan 17 16:56:36.611 [rsSync] oplog sync 3 of 3 m31003| Thu Jan 17 16:56:36.612 [rsSync] replSet initial sync finishing up m31002| Thu Jan 17 16:56:36.748 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:56:36.748 [initandlisten] connection accepted from 127.0.0.4:51134 #14 (9 connections now open) m31003| Thu Jan 17 16:56:36.799 [rsSync] replSet set minValid=50f87379:1 m31003| Thu Jan 17 16:56:36.800 [rsSync] build index local.replset.minvalid { _id: 1 } m31003| Thu Jan 17 16:56:36.800 [rsSync] build index done. scanned 0 total records. 0 secs m31003| Thu Jan 17 16:56:36.801 [rsSync] replSet initial sync done m31000| Thu Jan 17 16:56:36.801 [conn13] end connection 127.0.0.5:48716 (8 connections now open) m31001| Thu Jan 17 16:56:36.832 [rsSync] replSet SECONDARY m31000| Thu Jan 17 16:56:36.833 [slaveTracking] build index local.slaves { _id: 1 } m31000| Thu Jan 17 16:56:36.834 [slaveTracking] build index done. scanned 0 total records. 0 secs m31003| Thu Jan 17 16:56:36.843 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:36.843 [initandlisten] connection accepted from 127.0.0.5:55524 #15 (9 connections now open) m31002| Thu Jan 17 16:56:37.554 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:56:37.749 [rsSync] replSet SECONDARY m31003| Thu Jan 17 16:56:37.832 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:56:37.833 [initandlisten] connection accepted from 127.0.0.5:51127 #16 (10 connections now open) m31003| Thu Jan 17 16:56:38.179 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 16:56:38.180 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 16:56:38.284 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:56:38.285 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:56:38.801 [rsSync] replSet SECONDARY m31001| Thu Jan 17 16:56:38.903 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:56:38.903 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY ---- Started set! ---- { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 2 } } m31000| Thu Jan 17 16:56:38.920 [conn1] replSet replSetReconfig config object parses ok, 4 members specified m31000| Thu Jan 17 16:56:38.920 [conn1] replSet replSetReconfig [2] m31000| Thu Jan 17 16:56:38.920 [conn1] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:56:38.954 [conn1] replSet saveConfigLocally done m31000| Thu Jan 17 16:56:38.954 [conn1] replSet relinquishing primary state m31000| Thu Jan 17 16:56:38.954 [conn1] replSet SECONDARY m31000| Thu Jan 17 16:56:38.954 [conn1] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:56:38.955 [conn1] replSet PRIMARY Thu Jan 17 16:56:38.955 DBClientCursor::init call() failed m31002| Thu Jan 17 16:56:38.955 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31001| Thu Jan 17 16:56:38.955 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31003| Thu Jan 17 16:56:38.955 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:38.955 [conn1] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:56:38.955 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31001| Thu Jan 17 16:56:38.955 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:38.955 [conn9] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:33665] m31000| Thu Jan 17 16:56:38.955 [conn16] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.5:51127] m31000| Thu Jan 17 16:56:38.955 [conn1] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56467] m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:56:38.955 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:41208] m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:56:38.955 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31000| Thu Jan 17 16:56:38.955 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Thu Jan 17 16:56:38.955 [conn12] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:51989] m31000| Thu Jan 17 16:56:38.955 [conn15] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.5:55524] m31000| Thu Jan 17 16:56:38.955 [conn14] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:51134] Thu Jan 17 16:56:38.955 trying reconnect to 127.0.0.2:31000 Thu Jan 17 16:56:38.955 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:56:38.956 [initandlisten] connection accepted from 127.0.0.1:56497 #17 (4 connections now open) Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Thu Jan 17 16:56:38.961 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:11 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dst 127.0.0.5 flowid 1:11 > /tmp/output Thu Jan 17 16:56:38.974 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.2 match ip dst 127.0.0.5 flowid 1:11 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Thu Jan 17 16:56:38.986 shell: started program sudo bash -c tc qdisc add dev lo parent 1:11 handle 11: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 40000ms > /tmp/output Thu Jan 17 16:56:38.997 shell: started program sudo bash -c tc qdisc change dev lo parent 1:11 handle 11: netem delay 40000ms > /tmp/output Running: sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Thu Jan 17 16:56:39.011 shell: started program sudo bash -c tc class add dev lo parent 1:1 classid 1:12 htb rate 1bps ceil 1000Mbps > /tmp/output Running: sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dst 127.0.0.2 flowid 1:12 > /tmp/output Thu Jan 17 16:56:39.022 shell: started program sudo bash -c tc filter add dev lo parent 1: prio 1 u32 match ip src 127.0.0.5 match ip dst 127.0.0.2 flowid 1:12 > /tmp/output Running: sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Thu Jan 17 16:56:39.035 shell: started program sudo bash -c tc qdisc add dev lo parent 1:12 handle 12: netem > /tmp/output Running: sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output Thu Jan 17 16:56:39.050 shell: started program sudo bash -c tc qdisc change dev lo parent 1:12 handle 12: netem delay 40000ms > /tmp/output Running: sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output Thu Jan 17 16:56:39.064 shell: started program sudo bash -c tc -s -d -iec qdisc show dev lo > /tmp/output qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 2087 ver 3.17 Sent 362034 bytes 2088 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 11: parent 1:11 limit 1000 delay 40.0s Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 12: parent 1:12 limit 1000 delay 40.0s Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 3 } } m31000| Thu Jan 17 16:56:39.080 [conn17] replSet replSetReconfig config object parses ok, 4 members specified m31002| Thu Jan 17 16:56:39.554 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:56:39.554 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31002| Thu Jan 17 16:56:39.554 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:56:39.586 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:56:39.586 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:56:39.587 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:56:40.180 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31003| Thu Jan 17 16:56:40.180 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:56:40.181 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:56:40.181 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:56:40.181 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:56:40.181 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:56:40.181 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:56:40.181 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:56:40.284 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:56:40.285 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:56:40.285 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:40.904 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31001| Thu Jan 17 16:56:40.904 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:56:40.936 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:56:40.936 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:56:40.936 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31001| Thu Jan 17 16:56:40.936 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31001| Thu Jan 17 16:56:40.936 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31001| Thu Jan 17 16:56:40.936 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 2 2 m31001| Thu Jan 17 16:56:40.936 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:56:40.936 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:56:40.936 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:56:40.936 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 16:56:40.937 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:56:40.937 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31002| Thu Jan 17 16:56:40.955 [conn5] end connection 127.0.0.2:47048 (3 connections now open) m31002| Thu Jan 17 16:56:40.955 [initandlisten] connection accepted from 127.0.0.2:34332 #7 (4 connections now open) m31002| Thu Jan 17 16:56:41.554 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:41.554 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:41.554 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:41.587 [conn6] end connection 127.0.0.4:39510 (3 connections now open) m31001| Thu Jan 17 16:56:41.587 [initandlisten] connection accepted from 127.0.0.4:60562 #8 (4 connections now open) m31003| Thu Jan 17 16:56:42.180 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:56:42.180 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:42.181 [conn7] end connection 127.0.0.5:46858 (3 connections now open) m31001| Thu Jan 17 16:56:42.182 [initandlisten] connection accepted from 127.0.0.5:60891 #9 (4 connections now open) m31001| Thu Jan 17 16:56:42.904 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:42.904 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:42.904 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:42.937 [conn6] end connection 127.0.0.3:45160 (3 connections now open) m31002| Thu Jan 17 16:56:42.937 [initandlisten] connection accepted from 127.0.0.3:33595 #8 (4 connections now open) m31001| Thu Jan 17 16:56:48.955 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31002| Thu Jan 17 16:56:48.955 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:48.955 [initandlisten] connection accepted from 127.0.0.3:55454 #18 (5 connections now open) m31000| Thu Jan 17 16:56:48.955 [initandlisten] connection accepted from 127.0.0.4:42746 #19 (6 connections now open) m31001| Thu Jan 17 16:56:48.956 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31002| Thu Jan 17 16:56:48.956 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:56:48.956 [initandlisten] connection accepted from 127.0.0.3:57114 #20 (7 connections now open) m31000| Thu Jan 17 16:56:48.957 [initandlisten] connection accepted from 127.0.0.4:53396 #21 (8 connections now open) m31000| Thu Jan 17 16:56:49.079 [conn17] DBClientCursor::init call() failed m31000| Thu Jan 17 16:56:49.079 [conn17] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 10276 DBClientBase::findN: transport error: 127.0.0.5:31003 ns: admin.$cmd query: { replSetHeartbeat: "testReplSet", v: -1, pv: 1, checkEmpty: false, from: "" } m31000| Thu Jan 17 16:56:49.079 [conn17] replSet replSetReconfig [2] m31000| Thu Jan 17 16:56:49.079 [conn17] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:56:49.080 [conn17] replSet saveConfigLocally done m31000| Thu Jan 17 16:56:49.080 [conn17] replSet relinquishing primary state m31000| Thu Jan 17 16:56:49.080 [conn17] replSet SECONDARY m31000| Thu Jan 17 16:56:49.080 [conn17] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:56:49.081 [conn17] replSet PRIMARY Thu Jan 17 16:56:49.081 DBClientCursor::init call() failed m31001| Thu Jan 17 16:56:49.081 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31001| Thu Jan 17 16:56:49.081 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31002| Thu Jan 17 16:56:49.081 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:49.081 [conn18] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:55454] m31000| Thu Jan 17 16:56:49.081 [conn17] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 16:56:49.081 [conn17] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 0.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 40.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 3.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:1228 reslen:71 10000ms m31002| Thu Jan 17 16:56:49.081 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:49.081 [conn19] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:42746] m31000| Thu Jan 17 16:56:49.081 [conn17] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56497] m31000| Thu Jan 17 16:56:49.081 [conn21] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:53396] m31000| Thu Jan 17 16:56:49.081 [conn20] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:57114] m31000| Thu Jan 17 16:56:49.081 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:56:49.081 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:56:49.081 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:56:49.081 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 16:56:49.081 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Thu Jan 17 16:56:49.081 [rsMgr] replSet relinquishing primary state m31000| Thu Jan 17 16:56:49.081 [rsMgr] replSet SECONDARY m31000| Thu Jan 17 16:56:49.081 [rsMgr] replSet closing client sockets after relinquishing primary Thu Jan 17 16:56:49.082 trying reconnect to 127.0.0.2:31000 Thu Jan 17 16:56:49.082 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:56:49.082 [initandlisten] connection accepted from 127.0.0.1:56498 #22 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:49Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 44, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 43, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 10, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 42, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:49Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 42, "optime" : { "t" : 1358459769000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:09Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true } ], "ok" : 1 } Syncing to: undefined, it : 4, expected : 127.0.0.3:31001 m31002| Thu Jan 17 16:56:49.589 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:56:49.589 [rsMgr] replset msgReceivedNewConfig version: version: 3 m31002| Thu Jan 17 16:56:49.589 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:56:49.590 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:56:49.590 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:56:49.591 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31003| Thu Jan 17 16:56:50.179 [rsHealthPoll] DBClientCursor::init call() failed m31003| Thu Jan 17 16:56:50.179 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31003| Thu Jan 17 16:56:50.183 [rsMgr] replset msgReceivedNewConfig version: version: 3 m31003| Thu Jan 17 16:56:50.183 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:56:50.184 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:56:50.184 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:56:50.185 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:56:50.185 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 16:56:50.185 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:56:50.185 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 16:56:50.938 [rsMgr] replset msgReceivedNewConfig version: version: 3 m31001| Thu Jan 17 16:56:50.938 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:56:50.938 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Thu Jan 17 16:56:50.955 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31001| Thu Jan 17 16:56:50.955 [initandlisten] connection accepted from 127.0.0.5:49939 #10 (5 connections now open) m31000| Thu Jan 17 16:56:50.957 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:56:50.957 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:50.981 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:56:50.981 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:56:50.981 [rsMgr] replset msgReceivedNewConfig version: version: 3 m31001| Thu Jan 17 16:56:50.981 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 3 3 m31001| Thu Jan 17 16:56:50.981 [rsMgr] replset msgReceivedNewConfig version: version: 3 m31001| Thu Jan 17 16:56:50.982 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 3 3 m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:56:50.982 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 16:56:50.982 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31001| Thu Jan 17 16:56:50.983 [initandlisten] connection accepted from 127.0.0.5:37943 #11 (6 connections now open) m31000| Thu Jan 17 16:56:50.984 [initandlisten] connection accepted from 127.0.0.3:48571 #23 (5 connections now open) m31001| Thu Jan 17 16:56:50.984 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:56:51.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:56:51.180 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:56:51.180 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:56:51.180 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:51.589 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:51.589 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:56:51.589 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:51.984 [slaveTracking] build index local.slaves { _id: 1 } m31001| Thu Jan 17 16:56:51.985 [slaveTracking] build index done. scanned 0 total records. 0 secs m31003| Thu Jan 17 16:56:52.183 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:56:52.183 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:52.938 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:52.939 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:52.939 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:56:53.592 [conn4] end connection 127.0.0.4:54529 (3 connections now open) m31003| Thu Jan 17 16:56:53.592 [initandlisten] connection accepted from 127.0.0.4:40758 #5 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:54Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 48, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:56:54Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:51Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:56:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:56:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 47, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.3:31001 m31002| Thu Jan 17 16:56:54.185 [conn4] end connection 127.0.0.5:56462 (3 connections now open) m31002| Thu Jan 17 16:56:54.185 [initandlisten] connection accepted from 127.0.0.5:54351 #9 (4 connections now open) m31003| Thu Jan 17 16:56:54.982 [conn3] end connection 127.0.0.3:42339 (3 connections now open) m31003| Thu Jan 17 16:56:54.982 [initandlisten] connection accepted from 127.0.0.3:38520 #6 (5 connections now open) m31001| Thu Jan 17 16:56:55.082 [conn3] end connection 127.0.0.2:59723 (5 connections now open) m31001| Thu Jan 17 16:56:55.082 [initandlisten] connection accepted from 127.0.0.2:35730 #12 (6 connections now open) m31000| Thu Jan 17 16:56:55.592 [conn6] end connection 127.0.0.4:57313 (4 connections now open) m31000| Thu Jan 17 16:56:55.592 [initandlisten] connection accepted from 127.0.0.4:39998 #24 (5 connections now open) m31000| Thu Jan 17 16:56:56.983 [conn5] end connection 127.0.0.3:49790 (4 connections now open) m31000| Thu Jan 17 16:56:56.983 [initandlisten] connection accepted from 127.0.0.3:49120 #25 (5 connections now open) m31000| Thu Jan 17 16:56:59.079 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:56:59.079 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31000| Thu Jan 17 16:56:59.079 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:56:59.081 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31002| Thu Jan 17 16:56:59.081 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:56:59.081 [initandlisten] connection accepted from 127.0.0.4:35397 #26 (6 connections now open) m31000| Thu Jan 17 16:56:59.081 [initandlisten] connection accepted from 127.0.0.3:60230 #27 (7 connections now open) m31001| Thu Jan 17 16:56:59.082 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:56:59.082 [initandlisten] connection accepted from 127.0.0.3:59159 #28 (8 connections now open) m31003| Thu Jan 17 16:57:03.179 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31003| Thu Jan 17 16:57:08.188 [rsMgr] replSet I don't see a primary and I can't elect myself m31000| Thu Jan 17 16:57:09.079 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31002| Thu Jan 17 16:57:09.084 [conn7] end connection 127.0.0.2:34332 (3 connections now open) m31002| Thu Jan 17 16:57:09.085 [initandlisten] connection accepted from 127.0.0.2:34873 #10 (4 connections now open) m31001| Thu Jan 17 16:57:09.594 [conn8] end connection 127.0.0.4:60562 (5 connections now open) m31001| Thu Jan 17 16:57:09.594 [initandlisten] connection accepted from 127.0.0.4:36649 #13 (6 connections now open) m31000| Thu Jan 17 16:57:10.079 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Thu Jan 17 16:57:10.079 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:57:10.079 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31000| Thu Jan 17 16:57:10.080 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31000| Thu Jan 17 16:57:10.080 [rsMgr] replSet info electSelf 0 m31001| Thu Jan 17 16:57:10.080 [conn12] replSet info voting yea for 127.0.0.2:31000 (0) m31002| Thu Jan 17 16:57:10.080 [conn10] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Thu Jan 17 16:57:10.188 [conn9] end connection 127.0.0.5:60891 (5 connections now open) m31001| Thu Jan 17 16:57:10.188 [initandlisten] connection accepted from 127.0.0.5:52334 #14 (6 connections now open) m31000| Thu Jan 17 16:57:10.286 [rsMgr] replSet PRIMARY m31002| Thu Jan 17 16:57:10.986 [conn8] end connection 127.0.0.3:33595 (3 connections now open) m31001| Thu Jan 17 16:57:10.986 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:57:10.986 [initandlisten] connection accepted from 127.0.0.3:44046 #11 (5 connections now open) m31002| Thu Jan 17 16:57:11.595 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 4 } } m31000| Thu Jan 17 16:57:12.129 [conn22] replSet replSetReconfig config object parses ok, 4 members specified m31003| Thu Jan 17 16:57:16.179 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31000| Thu Jan 17 16:57:22.079 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:57:23.079 [conn22] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [5] server [127.0.0.5:31003] m31000| Thu Jan 17 16:57:23.079 [conn22] replSet replSetReconfig [2] m31000| Thu Jan 17 16:57:23.079 [conn22] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:57:23.089 [conn22] replSet saveConfigLocally done m31000| Thu Jan 17 16:57:23.089 [conn22] replSet relinquishing primary state m31000| Thu Jan 17 16:57:23.089 [conn22] replSet SECONDARY m31000| Thu Jan 17 16:57:23.089 [conn22] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:57:23.089 [conn22] replSet PRIMARY m31000| Thu Jan 17 16:57:23.089 [conn22] replSet replSetReconfig new config saved locally Thu Jan 17 16:57:23.089 DBClientCursor::init call() failed m31002| Thu Jan 17 16:57:23.089 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. Thu Jan 17 16:57:23.090 trying reconnect to 127.0.0.2:31000 m31001| Thu Jan 17 16:57:23.089 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:57:23.089 [conn22] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 40.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 0.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 4.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:9808 reslen:71 10960ms m31000| Thu Jan 17 16:57:23.089 [conn27] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:60230] m31000| Thu Jan 17 16:57:23.089 [conn22] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56498] m31000| Thu Jan 17 16:57:23.089 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:57:23.089 [conn26] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:35397] m31000| Thu Jan 17 16:57:23.089 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:57:23.089 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:57:23.090 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:57:23.090 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 16:57:23.090 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:57:23.090 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN Thu Jan 17 16:57:23.090 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:57:23.090 [initandlisten] connection accepted from 127.0.0.1:56500 #29 (6 connections now open) m31000| Thu Jan 17 16:57:23.091 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.ns, filling with zeroes... m31000| Thu Jan 17 16:57:23.239 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.ns, size: 16MB, took 0.148 secs m31000| Thu Jan 17 16:57:23.239 [FileAllocator] allocating new datafile /data/db/testReplSet-0/foo.0, filling with zeroes... m31000| Thu Jan 17 16:57:23.382 [FileAllocator] done allocating datafile /data/db/testReplSet-0/foo.0, size: 16MB, took 0.142 secs m31000| Thu Jan 17 16:57:23.383 [conn29] build index foo.bar { _id: 1 } m31000| Thu Jan 17 16:57:23.384 [conn29] build index done. scanned 0 total records. 0 secs m31000| Thu Jan 17 16:57:23.384 [conn29] insert foo.bar ninserted:1 keyUpdates:0 locks(micros) w:293524 293ms { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:23Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 78, "optime" : { "t" : 1358459843000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 33, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 77, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:23Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 34, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:22Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:21Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:22Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:23Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:16Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:22Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:22Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 76, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 16:57:23.597 [conn5] end connection 127.0.0.4:40758 (3 connections now open) m31002| Thu Jan 17 16:57:23.597 [rsMgr] replset msgReceivedNewConfig version: version: 4 m31002| Thu Jan 17 16:57:23.597 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:57:23.597 [initandlisten] connection accepted from 127.0.0.4:55787 #7 (4 connections now open) m31002| Thu Jan 17 16:57:23.598 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:57:23.598 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:57:23.599 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:57:24.191 [conn9] end connection 127.0.0.5:54351 (3 connections now open) m31002| Thu Jan 17 16:57:24.191 [initandlisten] connection accepted from 127.0.0.5:49607 #12 (4 connections now open) m31003| Thu Jan 17 16:57:24.192 [rsMgr] replset msgReceivedNewConfig version: version: 4 m31003| Thu Jan 17 16:57:24.192 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:57:24.193 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:57:24.193 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:57:24.193 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:57:24.193 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:57:24.193 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:57:24.193 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 16:57:24.988 [conn6] end connection 127.0.0.3:38520 (3 connections now open) m31003| Thu Jan 17 16:57:24.988 [initandlisten] connection accepted from 127.0.0.3:58333 #8 (4 connections now open) m31001| Thu Jan 17 16:57:24.988 [rsMgr] replset msgReceivedNewConfig version: version: 4 m31001| Thu Jan 17 16:57:24.988 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:57:24.989 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:57:24.989 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:57:24.989 [rsMgr] replset msgReceivedNewConfig version: version: 4 m31001| Thu Jan 17 16:57:24.989 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 4 4 m31001| Thu Jan 17 16:57:24.990 [rsMgr] replset msgReceivedNewConfig version: version: 4 m31001| Thu Jan 17 16:57:24.990 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 4 4 m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:57:24.990 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Thu Jan 17 16:57:25.079 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:57:25.087 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:57:25.087 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:25.597 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:25.598 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:25.598 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:57:26.191 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:57:26.192 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:26.988 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:26.988 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:26.989 [rsHealthPoll] Client::shutdown not called: rsHealthPoll { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 83, "optime" : { "t" : 1358459848000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358459843000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:27Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 82, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:28Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 5, "optime" : { "t" : 1358459843000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:28Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:28Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459798000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 81, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 16:57:29.079 [conn2] end connection 127.0.0.2:60206 (3 connections now open) m31002| Thu Jan 17 16:57:29.082 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:57:29.082 [initandlisten] connection accepted from 127.0.0.4:32988 #30 (7 connections now open) m31003| Thu Jan 17 16:57:29.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 16:57:30.179 [conn7] end connection 127.0.0.5:43645 (6 connections now open) m31002| Thu Jan 17 16:57:31.091 [conn10] end connection 127.0.0.2:34873 (3 connections now open) m31002| Thu Jan 17 16:57:31.091 [initandlisten] connection accepted from 127.0.0.2:58478 #13 (4 connections now open) m31001| Thu Jan 17 16:57:31.600 [conn13] end connection 127.0.0.4:36649 (5 connections now open) m31001| Thu Jan 17 16:57:31.600 [initandlisten] connection accepted from 127.0.0.4:39213 #15 (6 connections now open) m31001| Thu Jan 17 16:57:32.195 [conn14] end connection 127.0.0.5:52334 (5 connections now open) m31001| Thu Jan 17 16:57:32.195 [initandlisten] connection accepted from 127.0.0.5:43457 #16 (6 connections now open) m31002| Thu Jan 17 16:57:32.991 [conn11] end connection 127.0.0.3:44046 (3 connections now open) m31002| Thu Jan 17 16:57:32.991 [initandlisten] connection accepted from 127.0.0.3:35038 #14 (4 connections now open) m31002| Thu Jan 17 16:57:33.089 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31001| Thu Jan 17 16:57:33.090 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:57:33.090 [initandlisten] connection accepted from 127.0.0.4:44020 #31 (7 connections now open) m31000| Thu Jan 17 16:57:33.090 [initandlisten] connection accepted from 127.0.0.3:52707 #32 (8 connections now open) m31002| Thu Jan 17 16:57:33.091 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:57:33.091 [conn30] end connection 127.0.0.4:32988 (7 connections now open) m31000| Thu Jan 17 16:57:33.091 [initandlisten] connection accepted from 127.0.0.4:42261 #33 (8 connections now open) m31002| Thu Jan 17 16:57:33.092 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.ns, filling with zeroes... m31002| Thu Jan 17 16:57:33.267 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.ns, size: 16MB, took 0.174 secs m31002| Thu Jan 17 16:57:33.267 [FileAllocator] allocating new datafile /data/db/testReplSet-2/foo.0, filling with zeroes... { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 88, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459843000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459848000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 87, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } m31002| Thu Jan 17 16:57:33.417 [FileAllocator] done allocating datafile /data/db/testReplSet-2/foo.0, size: 16MB, took 0.149 secs m31002| Thu Jan 17 16:57:33.419 [repl writer worker 1] build index foo.bar { _id: 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 10, "optime" : { "t" : 1358459848000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:33Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 86, "optime" : { "t" : 1358459843000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } m31002| Thu Jan 17 16:57:33.420 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:33Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 86, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31000| Thu Jan 17 16:57:35.087 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 93, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:37Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 92, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:38Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 15, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 91, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:38Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:38Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459853000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 91, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 16:57:39.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 16:57:40.094 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31001| Thu Jan 17 16:57:40.094 [conn10] end connection 127.0.0.5:49939 (5 connections now open) m31002| Thu Jan 17 16:57:40.094 [initandlisten] connection accepted from 127.0.0.5:39681 #15 (5 connections now open) m31003| Thu Jan 17 16:57:40.095 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Thu Jan 17 16:57:40.095 [conn11] end connection 127.0.0.5:37943 (4 connections now open) m31002| Thu Jan 17 16:57:40.095 [initandlisten] connection accepted from 127.0.0.5:53803 #16 (6 connections now open) m31003| Thu Jan 17 16:57:40.096 [FileAllocator] allocating new datafile /data/db/testReplSet-3/foo.ns, filling with zeroes... m31003| Thu Jan 17 16:57:40.246 [FileAllocator] done allocating datafile /data/db/testReplSet-3/foo.ns, size: 16MB, took 0.149 secs m31003| Thu Jan 17 16:57:40.246 [FileAllocator] allocating new datafile /data/db/testReplSet-3/foo.0, filling with zeroes... m31003| Thu Jan 17 16:57:40.396 [FileAllocator] done allocating datafile /data/db/testReplSet-3/foo.0, size: 16MB, took 0.149 secs m31003| Thu Jan 17 16:57:40.398 [repl writer worker 1] build index foo.bar { _id: 1 } m31003| Thu Jan 17 16:57:40.398 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31002| Thu Jan 17 16:57:40.439 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Thu Jan 17 16:57:40.439 [initandlisten] connection accepted from 127.0.0.4:41635 #34 (9 connections now open) m31002| Thu Jan 17 16:57:41.440 [slaveTracking] build index local.slaves { _id: 1 } m31002| Thu Jan 17 16:57:41.440 [slaveTracking] build index done. scanned 0 total records. 0 secs { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 98, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 97, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 20, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:43Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 96, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:43Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 96, "optime" : { "t" : 1358459858000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:38Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.4:31002 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 5 } } m31000| Thu Jan 17 16:57:43.447 [conn29] replSet replSetReconfig config object parses ok, 4 members specified m31003| Thu Jan 17 16:57:45.603 [conn7] end connection 127.0.0.4:55787 (2 connections now open) m31003| Thu Jan 17 16:57:45.603 [initandlisten] connection accepted from 127.0.0.4:48153 #9 (3 connections now open) m31002| Thu Jan 17 16:57:46.198 [conn12] end connection 127.0.0.5:49607 (5 connections now open) m31002| Thu Jan 17 16:57:46.198 [initandlisten] connection accepted from 127.0.0.5:57692 #17 (6 connections now open) m31003| Thu Jan 17 16:57:46.994 [conn8] end connection 127.0.0.3:58333 (2 connections now open) m31003| Thu Jan 17 16:57:46.994 [initandlisten] connection accepted from 127.0.0.3:36255 #10 (4 connections now open) m31001| Thu Jan 17 16:57:47.094 [conn12] end connection 127.0.0.2:35730 (3 connections now open) m31001| Thu Jan 17 16:57:47.094 [initandlisten] connection accepted from 127.0.0.2:46940 #17 (4 connections now open) m31000| Thu Jan 17 16:57:47.603 [conn24] end connection 127.0.0.4:39998 (8 connections now open) m31000| Thu Jan 17 16:57:47.603 [initandlisten] connection accepted from 127.0.0.4:52321 #35 (9 connections now open) m31000| Thu Jan 17 16:57:48.087 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:57:48.994 [conn25] end connection 127.0.0.3:49120 (8 connections now open) m31000| Thu Jan 17 16:57:48.994 [initandlisten] connection accepted from 127.0.0.3:34720 #36 (9 connections now open) m31000| Thu Jan 17 16:57:49.087 [conn29] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [5] server [127.0.0.5:31003] m31000| Thu Jan 17 16:57:49.087 [conn29] replSet replSetReconfig [2] m31000| Thu Jan 17 16:57:49.087 [conn29] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:57:49.125 [conn29] replSet saveConfigLocally done m31000| Thu Jan 17 16:57:49.125 [conn29] replSet relinquishing primary state m31000| Thu Jan 17 16:57:49.125 [conn29] replSet SECONDARY m31000| Thu Jan 17 16:57:49.125 [conn29] replSet closing client sockets after relinquishing primary Thu Jan 17 16:57:49.125 DBClientCursor::init call() failed m31000| Thu Jan 17 16:57:49.125 [conn29] replSet PRIMARY m31000| Thu Jan 17 16:57:49.125 [conn29] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 16:57:49.125 [conn34] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:41635] m31001| Thu Jan 17 16:57:49.125 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:57:49.125 [conn29] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 0.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 40.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 5.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:37684 reslen:71 5678ms Caught exception error doing query: failed, this is normal after reconfig. m31002| Thu Jan 17 16:57:49.125 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:57:49.125 [conn31] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:44020] m31000| Thu Jan 17 16:57:49.125 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:57:49.125 [conn29] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56500] m31000| Thu Jan 17 16:57:49.125 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:57:49.125 [conn32] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:52707] m31000| Thu Jan 17 16:57:49.125 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:57:49.125 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:57:49.125 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Thu Jan 17 16:57:49.125 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:57:49.125 [rsMgr] replSet relinquishing primary state m31000| Thu Jan 17 16:57:49.125 [rsMgr] replSet SECONDARY m31000| Thu Jan 17 16:57:49.125 [rsMgr] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:57:49.126 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31000| Thu Jan 17 16:57:49.126 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY Thu Jan 17 16:57:49.126 trying reconnect to 127.0.0.2:31000 m31000| Thu Jan 17 16:57:49.126 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' Thu Jan 17 16:57:49.126 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:57:49.126 [initandlisten] connection accepted from 127.0.0.1:56501 #37 (6 connections now open) m31000| Thu Jan 17 16:57:49.126 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:49Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 104, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 103, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:49Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 26, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 102, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:49Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 102, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:57:49.179 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:49.604 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:57:49.604 [rsMgr] replset msgReceivedNewConfig version: version: 5 m31002| Thu Jan 17 16:57:49.604 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:57:49.639 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:57:49.639 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:57:49.639 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:57:49.639 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:57:49.639 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:57:49.639 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:57:49.640 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:57:49.640 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:57:49.640 [rsMgr] replSet I don't see a primary and I can't elect myself m31003| Thu Jan 17 16:57:50.199 [rsMgr] replset msgReceivedNewConfig version: version: 5 m31003| Thu Jan 17 16:57:50.199 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:57:50.218 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:57:50.218 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:57:50.219 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:57:50.219 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 16:57:50.219 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:57:50.219 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 16:57:50.995 [rsMgr] replset msgReceivedNewConfig version: version: 5 m31001| Thu Jan 17 16:57:50.995 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:57:50.995 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 16:57:50.996 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:57:50.996 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:57:50.996 [rsMgr] replset msgReceivedNewConfig version: version: 5 m31001| Thu Jan 17 16:57:50.996 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 5 5 m31001| Thu Jan 17 16:57:50.996 [rsMgr] replset msgReceivedNewConfig version: version: 5 m31001| Thu Jan 17 16:57:50.996 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 5 5 m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:57:50.996 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:57:50.996 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31000| Thu Jan 17 16:57:51.087 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:57:51.095 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:57:51.095 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:51.604 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:51.604 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:57:51.604 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:57:52.199 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:57:52.199 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:52.995 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:52.995 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:57:52.995 [rsHealthPoll] Client::shutdown not called: rsHealthPoll { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:54Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 109, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:53Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 108, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:54Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 107, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:54Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 107, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31000| Thu Jan 17 16:57:55.127 [rsMgr] replSet info electSelf 0 m31001| Thu Jan 17 16:57:55.127 [conn17] replSet info voting yea for 127.0.0.2:31000 (0) m31002| Thu Jan 17 16:57:55.127 [conn13] replSet info voting yea for 127.0.0.2:31000 (0) m31000| Thu Jan 17 16:57:55.291 [rsMgr] replSet PRIMARY m31002| Thu Jan 17 16:57:55.641 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 16:57:56.997 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:57:59.125 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:57:59.126 [initandlisten] connection accepted from 127.0.0.4:34511 #38 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:59Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 114, "optime" : { "t" : 1358459879000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:59Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 113, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:59Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 10, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 112, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:57:59Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:57:58Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:57:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 112, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:57:59.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 16:57:59.641 [conn9] end connection 127.0.0.4:48153 (2 connections now open) m31003| Thu Jan 17 16:57:59.641 [initandlisten] connection accepted from 127.0.0.4:38804 #11 (3 connections now open) m31002| Thu Jan 17 16:58:00.221 [conn17] end connection 127.0.0.5:57692 (5 connections now open) m31002| Thu Jan 17 16:58:00.221 [initandlisten] connection accepted from 127.0.0.5:53902 #18 (6 connections now open) m31003| Thu Jan 17 16:58:00.998 [conn10] end connection 127.0.0.3:36255 (2 connections now open) m31003| Thu Jan 17 16:58:00.998 [initandlisten] connection accepted from 127.0.0.3:40461 #12 (3 connections now open) m31000| Thu Jan 17 16:58:01.123 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31001| Thu Jan 17 16:58:01.128 [conn17] end connection 127.0.0.2:46940 (3 connections now open) m31001| Thu Jan 17 16:58:01.128 [initandlisten] connection accepted from 127.0.0.2:33913 #18 (4 connections now open) m31000| Thu Jan 17 16:58:01.642 [conn35] end connection 127.0.0.4:52321 (6 connections now open) m31000| Thu Jan 17 16:58:01.642 [initandlisten] connection accepted from 127.0.0.4:44102 #39 (7 connections now open) m31000| Thu Jan 17 16:58:02.998 [conn36] end connection 127.0.0.3:34720 (6 connections now open) m31000| Thu Jan 17 16:58:02.998 [initandlisten] connection accepted from 127.0.0.3:36668 #40 (7 connections now open) m31001| Thu Jan 17 16:58:03.091 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:58:03.091 [conn28] end connection 127.0.0.3:59159 (6 connections now open) m31000| Thu Jan 17 16:58:03.091 [initandlisten] connection accepted from 127.0.0.3:56821 #41 (7 connections now open) m31001| Thu Jan 17 16:58:03.124 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.ns, filling with zeroes... m31001| Thu Jan 17 16:58:03.266 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.ns, size: 16MB, took 0.141 secs m31001| Thu Jan 17 16:58:03.266 [FileAllocator] allocating new datafile /data/db/testReplSet-1/foo.0, filling with zeroes... m31001| Thu Jan 17 16:58:03.407 [FileAllocator] done allocating datafile /data/db/testReplSet-1/foo.0, size: 16MB, took 0.141 secs m31001| Thu Jan 17 16:58:03.409 [repl writer worker 1] build index foo.bar { _id: 1 } m31001| Thu Jan 17 16:58:03.409 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Thu Jan 17 16:58:03.410 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:58:03.410 [initandlisten] connection accepted from 127.0.0.3:55887 #42 (8 connections now open) m31001| Thu Jan 17 16:58:03.441 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:58:03.441 [conn41] end connection 127.0.0.3:56821 (7 connections now open) m31000| Thu Jan 17 16:58:03.442 [initandlisten] connection accepted from 127.0.0.3:52368 #43 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:04Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 119, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459843000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:23Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459879000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:59Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:03Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 118, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:04Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 15, "optime" : { "t" : 1358459879000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:59Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459879000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:59Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:04Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459809000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:56:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:02Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 117, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:09Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 124, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 123, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:09Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 20, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:08Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 122, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:08Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:09Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459884000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:04Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:08Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 122, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:58:09.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 16:58:14.123 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:14Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 129, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:01Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 24, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:13Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 128, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:14Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 25, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 127, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:13Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:14Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459889000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:09Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:12Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:12Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 127, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31002| Thu Jan 17 16:58:15.130 [conn13] end connection 127.0.0.2:58478 (5 connections now open) m31002| Thu Jan 17 16:58:15.130 [initandlisten] connection accepted from 127.0.0.2:44450 #19 (6 connections now open) m31001| Thu Jan 17 16:58:15.644 [conn15] end connection 127.0.0.4:39213 (3 connections now open) m31001| Thu Jan 17 16:58:15.644 [initandlisten] connection accepted from 127.0.0.4:59160 #19 (4 connections now open) m31001| Thu Jan 17 16:58:16.224 [conn16] end connection 127.0.0.5:43457 (3 connections now open) m31001| Thu Jan 17 16:58:16.224 [initandlisten] connection accepted from 127.0.0.5:56892 #20 (4 connections now open) m31002| Thu Jan 17 16:58:17.001 [conn14] end connection 127.0.0.3:35038 (5 connections now open) m31002| Thu Jan 17 16:58:17.001 [initandlisten] connection accepted from 127.0.0.3:60944 #20 (6 connections now open) m31003| Thu Jan 17 16:58:19.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:58:19.179 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:58:19.179 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:58:19.179 [rsMgr] replSet I don't see a primary and I can't elect myself { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 134, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 29, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 133, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:19Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 30, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:18Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 132, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:17Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:18Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:19Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459894000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:18Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:18Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 132, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:58:24.195 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Thu Jan 17 16:58:24.195 [conn15] end connection 127.0.0.5:39681 (5 connections now open) m31001| Thu Jan 17 16:58:24.195 [initandlisten] connection accepted from 127.0.0.5:55495 #21 (5 connections now open) m31003| Thu Jan 17 16:58:24.196 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31002| Thu Jan 17 16:58:24.196 [conn16] end connection 127.0.0.5:53803 (4 connections now open) m31001| Thu Jan 17 16:58:24.196 [initandlisten] connection accepted from 127.0.0.5:40670 #22 (6 connections now open) m31001| Thu Jan 17 16:58:24.197 [rsGhostSync] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31001| Thu Jan 17 16:58:24.197 [rsGhostSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31000| Thu Jan 17 16:58:24.197 [conn23] end connection 127.0.0.3:48571 (7 connections now open) m31001| Thu Jan 17 16:58:24.197 [rsGhostSync] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31001| Thu Jan 17 16:58:24.197 [rsGhostSync] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:24Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 139, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:23Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 34, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:23Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 138, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:23Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:24Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 35, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 137, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:23Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:24Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:22Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:22Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:23Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 137, "optime" : { "t" : 1358459899000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:19Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.3:31001 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 6 } } m31000| Thu Jan 17 16:58:24.211 [conn37] replSet replSetReconfig config object parses ok, 4 members specified m31000| Thu Jan 17 16:58:25.200 [initandlisten] connection accepted from 127.0.0.3:49482 #44 (8 connections now open) m31001| Thu Jan 17 16:58:25.200 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Thu Jan 17 16:58:27.123 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:58:28.123 [conn37] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [6] server [127.0.0.5:31003] m31000| Thu Jan 17 16:58:28.123 [conn37] replSet replSetReconfig [2] m31000| Thu Jan 17 16:58:28.123 [conn37] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:58:28.146 [conn37] replSet saveConfigLocally done m31000| Thu Jan 17 16:58:28.146 [conn37] replSet relinquishing primary state m31000| Thu Jan 17 16:58:28.146 [conn37] replSet SECONDARY m31000| Thu Jan 17 16:58:28.146 [conn37] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:58:28.147 [conn37] replSet PRIMARY m31001| Thu Jan 17 16:58:28.147 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:58:28.147 [conn44] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:49482] m31002| Thu Jan 17 16:58:28.147 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Thu Jan 17 16:58:28.147 DBClientCursor::init call() failed m31001| Thu Jan 17 16:58:28.147 [conn18] end connection 127.0.0.2:33913 (5 connections now open) m31001| Thu Jan 17 16:58:28.147 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:58:28.147 [conn37] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 16:58:28.147 [conn43] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:52368] m31000| Thu Jan 17 16:58:28.147 [conn37] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 40.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 0.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 6.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:23205 reslen:71 3935ms m31000| Thu Jan 17 16:58:28.147 [conn42] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:55887] m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:58:28.147 [conn38] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:34511] m31000| Thu Jan 17 16:58:28.147 [conn37] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56501] Caught exception error doing query: failed, this is normal after reconfig. m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31000| Thu Jan 17 16:58:28.147 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Thu Jan 17 16:58:28.147 [rsMgr] replSet relinquishing primary state m31000| Thu Jan 17 16:58:28.147 [rsMgr] replSet SECONDARY m31000| Thu Jan 17 16:58:28.147 [rsMgr] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] couldn't connect to 127.0.0.3:31001: couldn't connect to server 127.0.0.3:31001 m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:58:28.147 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:58:28.147 [rsMgr] replSet can't see a majority, will not try to elect self Thu Jan 17 16:58:28.147 trying reconnect to 127.0.0.2:31000 m31001| Thu Jan 17 16:58:28.147 [initandlisten] connection accepted from 127.0.0.2:51197 #23 (6 connections now open) m31001| Thu Jan 17 16:58:28.147 [conn23] end connection 127.0.0.2:51197 (5 connections now open) m31001| Thu Jan 17 16:58:28.147 [initandlisten] connection accepted from 127.0.0.2:52882 #24 (7 connections now open) m31000| Thu Jan 17 16:58:28.148 [rsHealthPoll] replSet member 127.0.0.3:31001 is up Thu Jan 17 16:58:28.148 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:58:28.148 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 16:58:28.148 [initandlisten] connection accepted from 127.0.0.1:56504 #45 (4 connections now open) m31000| Thu Jan 17 16:58:28.148 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:28Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 143, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 38, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 39, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 141, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:28Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:26Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459863000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 141, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31001| Thu Jan 17 16:58:29.003 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 16:58:29.003 [rsMgr] replset msgReceivedNewConfig version: version: 6 m31001| Thu Jan 17 16:58:29.003 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:58:29.029 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:58:29.029 [rsMgr] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 16:58:29.029 [conn40] end connection 127.0.0.3:36668 (3 connections now open) m31001| Thu Jan 17 16:58:29.029 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:58:29.030 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:58:29.030 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:58:29.030 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:58:29.030 [initandlisten] connection accepted from 127.0.0.3:56134 #46 (4 connections now open) m31001| Thu Jan 17 16:58:29.030 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:58:29.030 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:58:29.127 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:58:29.127 [conn33] end connection 127.0.0.4:42261 (3 connections now open) m31000| Thu Jan 17 16:58:29.127 [initandlisten] connection accepted from 127.0.0.4:34682 #47 (4 connections now open) m31000| Thu Jan 17 16:58:29.133 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:58:29.133 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:58:29.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 16:58:29.647 [conn11] end connection 127.0.0.4:38804 (2 connections now open) m31002| Thu Jan 17 16:58:29.647 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Thu Jan 17 16:58:29.647 [initandlisten] connection accepted from 127.0.0.4:49029 #13 (3 connections now open) m31002| Thu Jan 17 16:58:29.647 [rsMgr] replset msgReceivedNewConfig version: version: 6 m31002| Thu Jan 17 16:58:29.647 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:58:29.648 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:58:29.648 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:58:29.648 [rsMgr] replset msgReceivedNewConfig version: version: 6 m31002| Thu Jan 17 16:58:29.648 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 6 6 m31000| Thu Jan 17 16:58:29.648 [conn39] end connection 127.0.0.4:44102 (3 connections now open) m31002| Thu Jan 17 16:58:29.648 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:58:29.648 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:58:29.648 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:58:29.648 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Thu Jan 17 16:58:29.648 [initandlisten] connection accepted from 127.0.0.4:35740 #48 (4 connections now open) m31002| Thu Jan 17 16:58:29.649 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:58:29.649 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31000| Thu Jan 17 16:58:30.123 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:58:30.226 [conn18] end connection 127.0.0.5:53902 (3 connections now open) m31002| Thu Jan 17 16:58:30.227 [initandlisten] connection accepted from 127.0.0.5:50973 #21 (4 connections now open) m31003| Thu Jan 17 16:58:30.227 [rsMgr] replset msgReceivedNewConfig version: version: 6 m31003| Thu Jan 17 16:58:30.227 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:58:30.248 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:58:30.249 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:58:30.249 [rsMgr] replset msgReceivedNewConfig version: version: 6 m31003| Thu Jan 17 16:58:30.249 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 6 6 m31003| Thu Jan 17 16:58:30.249 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:58:30.249 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:58:30.249 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:58:30.249 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:58:31.003 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:58:31.003 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:58:31.003 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:58:31.647 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:58:31.647 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:58:31.647 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:58:32.227 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:58:32.227 [rsHealthPoll] Client::shutdown not called: rsHealthPoll { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:33Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 148, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 147, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:33Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 146, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31000| Thu Jan 17 16:58:34.149 [rsMgr] replSet info electSelf 0 m31002| Thu Jan 17 16:58:34.149 [conn19] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Thu Jan 17 16:58:34.149 [conn24] replSet info voting yea for 127.0.0.2:31000 (0) m31000| Thu Jan 17 16:58:34.295 [rsMgr] replSet PRIMARY m31001| Thu Jan 17 16:58:35.031 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 16:58:35.650 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 16:58:38.147 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:58:38.147 [initandlisten] connection accepted from 127.0.0.3:38469 #49 (5 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 153, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 152, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 151, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:38Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459869000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:57:49Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:36Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 151, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31002| Thu Jan 17 16:58:39.128 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:58:39.128 [initandlisten] connection accepted from 127.0.0.4:46252 #50 (6 connections now open) m31002| Thu Jan 17 16:58:39.129 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:58:39.129 [conn47] end connection 127.0.0.4:34682 (5 connections now open) m31000| Thu Jan 17 16:58:39.130 [initandlisten] connection accepted from 127.0.0.4:41728 #51 (6 connections now open) m31003| Thu Jan 17 16:58:39.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 16:58:42.150 [conn19] end connection 127.0.0.2:44450 (3 connections now open) m31002| Thu Jan 17 16:58:42.150 [initandlisten] connection accepted from 127.0.0.2:52724 #22 (4 connections now open) m31002| Thu Jan 17 16:58:43.032 [conn20] end connection 127.0.0.3:60944 (3 connections now open) m31002| Thu Jan 17 16:58:43.032 [initandlisten] connection accepted from 127.0.0.3:52816 #23 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 158, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 157, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 156, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:43Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358459918000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 156, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31001| Thu Jan 17 16:58:43.651 [conn19] end connection 127.0.0.4:59160 (5 connections now open) m31001| Thu Jan 17 16:58:43.651 [initandlisten] connection accepted from 127.0.0.4:53419 #25 (6 connections now open) m31001| Thu Jan 17 16:58:44.251 [conn20] end connection 127.0.0.5:56892 (5 connections now open) m31001| Thu Jan 17 16:58:44.251 [initandlisten] connection accepted from 127.0.0.5:35228 #26 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 163, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 161, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:48Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:46Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 161, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 16:58:49.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:58:49.179 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:58:49.179 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:58:52.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 168, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 24, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 167, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 24, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 166, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:53Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 166, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 16:58:57.035 [conn12] end connection 127.0.0.3:40461 (2 connections now open) m31003| Thu Jan 17 16:58:57.035 [initandlisten] connection accepted from 127.0.0.3:45724 #14 (3 connections now open) m31003| Thu Jan 17 16:58:57.654 [conn13] end connection 127.0.0.4:49029 (2 connections now open) m31003| Thu Jan 17 16:58:57.654 [initandlisten] connection accepted from 127.0.0.4:57496 #15 (3 connections now open) m31001| Thu Jan 17 16:58:58.153 [conn24] end connection 127.0.0.2:52882 (5 connections now open) m31001| Thu Jan 17 16:58:58.153 [initandlisten] connection accepted from 127.0.0.2:52100 #27 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:58Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 173, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 29, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 172, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:58Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 29, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:58Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 171, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:57Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:58:58Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 28, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 28, "optime" : { "t" : 1358459933000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:56Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:58:57Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 171, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31002| Thu Jan 17 16:58:58.254 [conn21] end connection 127.0.0.5:50973 (3 connections now open) m31002| Thu Jan 17 16:58:58.254 [initandlisten] connection accepted from 127.0.0.5:36672 #24 (5 connections now open) m31000| Thu Jan 17 16:58:59.035 [conn46] end connection 127.0.0.3:56134 (5 connections now open) m31000| Thu Jan 17 16:58:59.036 [initandlisten] connection accepted from 127.0.0.3:46267 #52 (6 connections now open) m31003| Thu Jan 17 16:58:59.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 16:58:59.654 [conn48] end connection 127.0.0.4:35740 (5 connections now open) m31000| Thu Jan 17 16:58:59.654 [initandlisten] connection accepted from 127.0.0.4:35884 #53 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:03Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 178, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 34, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 177, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:03Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 34, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:02Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 176, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:01Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:02Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:03Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:02Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:03Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459938000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:58Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:02Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 176, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31000| Thu Jan 17 16:59:05.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31003| Thu Jan 17 16:59:05.289 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31001| Thu Jan 17 16:59:05.289 [conn21] end connection 127.0.0.5:55495 (5 connections now open) m31002| Thu Jan 17 16:59:05.289 [initandlisten] connection accepted from 127.0.0.5:42969 #25 (5 connections now open) m31003| Thu Jan 17 16:59:05.290 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Thu Jan 17 16:59:05.291 [conn22] end connection 127.0.0.5:40670 (4 connections now open) m31002| Thu Jan 17 16:59:05.291 [initandlisten] connection accepted from 127.0.0.5:33272 #26 (6 connections now open) m31000| Thu Jan 17 16:59:05.292 [initandlisten] connection accepted from 127.0.0.4:32972 #54 (7 connections now open) m31002| Thu Jan 17 16:59:05.292 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31001| Thu Jan 17 16:59:08.148 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:59:08.149 [initandlisten] connection accepted from 127.0.0.3:46639 #55 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:08Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 183, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 39, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 182, "optime" : { "t" : 1358459908000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:28Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:07Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:08Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 39, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:08Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 181, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:08Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459904000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:24Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:07Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:06Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:07Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 181, "optime" : { "t" : 1358459943000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:03Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.4:31002 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 7 } } m31000| Thu Jan 17 16:59:08.241 [conn45] replSet replSetReconfig config object parses ok, 4 members specified m31003| Thu Jan 17 16:59:09.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 16:59:12.156 [conn22] end connection 127.0.0.2:52724 (5 connections now open) m31002| Thu Jan 17 16:59:12.156 [initandlisten] connection accepted from 127.0.0.2:59360 #27 (6 connections now open) m31002| Thu Jan 17 16:59:13.038 [conn23] end connection 127.0.0.3:52816 (5 connections now open) m31002| Thu Jan 17 16:59:13.038 [initandlisten] connection accepted from 127.0.0.3:51462 #28 (6 connections now open) m31001| Thu Jan 17 16:59:13.657 [conn25] end connection 127.0.0.4:53419 (3 connections now open) m31001| Thu Jan 17 16:59:13.657 [initandlisten] connection accepted from 127.0.0.4:33681 #28 (4 connections now open) m31001| Thu Jan 17 16:59:14.257 [conn26] end connection 127.0.0.5:35228 (3 connections now open) m31001| Thu Jan 17 16:59:14.257 [initandlisten] connection accepted from 127.0.0.5:42471 #29 (4 connections now open) m31000| Thu Jan 17 16:59:18.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31003| Thu Jan 17 16:59:19.179 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 16:59:27.040 [conn14] end connection 127.0.0.3:45724 (2 connections now open) m31003| Thu Jan 17 16:59:27.041 [initandlisten] connection accepted from 127.0.0.3:45019 #16 (3 connections now open) m31003| Thu Jan 17 16:59:27.659 [conn15] end connection 127.0.0.4:57496 (2 connections now open) m31003| Thu Jan 17 16:59:27.660 [initandlisten] connection accepted from 127.0.0.4:39442 #17 (3 connections now open) m31000| Thu Jan 17 16:59:28.147 [conn45] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [6] server [127.0.0.5:31003] m31000| Thu Jan 17 16:59:28.147 [conn45] replSet replSetReconfig [2] m31000| Thu Jan 17 16:59:28.147 [conn45] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:59:28.159 [conn27] end connection 127.0.0.2:52100 (3 connections now open) m31001| Thu Jan 17 16:59:28.159 [initandlisten] connection accepted from 127.0.0.2:43899 #30 (4 connections now open) m31000| Thu Jan 17 16:59:28.202 [conn45] replSet saveConfigLocally done m31000| Thu Jan 17 16:59:28.202 [conn45] replSet relinquishing primary state m31000| Thu Jan 17 16:59:28.202 [conn45] replSet SECONDARY m31000| Thu Jan 17 16:59:28.202 [conn45] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:59:28.202 [conn45] replSet PRIMARY Thu Jan 17 16:59:28.203 DBClientCursor::init call() failed m31001| Thu Jan 17 16:59:28.203 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31000| Thu Jan 17 16:59:28.203 [conn45] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 16:59:28.203 [conn45] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 0.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 40.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 7.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:55194 reslen:71 19961ms m31000| Thu Jan 17 16:59:28.203 [conn50] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:46252] m31000| Thu Jan 17 16:59:28.203 [conn49] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:38469] m31002| Thu Jan 17 16:59:28.203 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:59:28.203 [conn45] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56504] m31000| Thu Jan 17 16:59:28.203 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:59:28.203 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:59:28.203 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:59:28.203 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY Thu Jan 17 16:59:28.203 trying reconnect to 127.0.0.2:31000 Thu Jan 17 16:59:28.204 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:59:28.204 [initandlisten] connection accepted from 127.0.0.1:56506 #56 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:28Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 203, "optime" : { "t" : 1358459968000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:59:28Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:28Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 59, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 202, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:28Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 59, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:28Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 59, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:27Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:28Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:58:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 58, "optime" : { "t" : 1358459923000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 58, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:26Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:27Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 201, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31002| Thu Jan 17 16:59:28.260 [conn24] end connection 127.0.0.5:36672 (5 connections now open) m31002| Thu Jan 17 16:59:28.260 [initandlisten] connection accepted from 127.0.0.5:43299 #29 (6 connections now open) m31000| Thu Jan 17 16:59:29.041 [conn52] end connection 127.0.0.3:46267 (5 connections now open) m31000| Thu Jan 17 16:59:29.041 [initandlisten] connection accepted from 127.0.0.3:55993 #57 (6 connections now open) m31001| Thu Jan 17 16:59:29.042 [rsMgr] replset msgReceivedNewConfig version: version: 7 m31001| Thu Jan 17 16:59:29.042 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:59:29.056 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:59:29.056 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:59:29.057 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31000| Thu Jan 17 16:59:29.147 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:59:29.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:59:29.147 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:59:29.147 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31003| Thu Jan 17 16:59:29.179 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 16:59:29.179 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 16:59:29.179 [rsMgr] replSet I don't see a primary and I can't elect myself m31000| Thu Jan 17 16:59:29.660 [conn53] end connection 127.0.0.4:35884 (5 connections now open) m31000| Thu Jan 17 16:59:29.660 [initandlisten] connection accepted from 127.0.0.4:50545 #58 (6 connections now open) m31002| Thu Jan 17 16:59:29.660 [rsMgr] replset msgReceivedNewConfig version: version: 7 m31002| Thu Jan 17 16:59:29.660 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:59:29.722 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:59:29.723 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:59:29.723 [rsMgr] replset msgReceivedNewConfig version: version: 7 m31002| Thu Jan 17 16:59:29.723 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 7 7 m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 16:59:29.723 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31000| Thu Jan 17 16:59:30.159 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:59:30.160 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:30.203 [conn27] end connection 127.0.0.2:59360 (5 connections now open) m31002| Thu Jan 17 16:59:30.203 [initandlisten] connection accepted from 127.0.0.2:52497 #30 (6 connections now open) m31003| Thu Jan 17 16:59:30.261 [rsMgr] replset msgReceivedNewConfig version: version: 7 m31003| Thu Jan 17 16:59:30.261 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:59:30.269 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:59:30.269 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:59:30.269 [rsMgr] replset msgReceivedNewConfig version: version: 7 m31003| Thu Jan 17 16:59:30.269 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 7 7 m31003| Thu Jan 17 16:59:30.269 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:59:30.269 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:59:30.269 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:59:30.269 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:59:31.041 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:31.041 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:31.042 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:31.057 [conn28] end connection 127.0.0.3:51462 (5 connections now open) m31002| Thu Jan 17 16:59:31.057 [initandlisten] connection accepted from 127.0.0.3:34926 #31 (6 connections now open) m31003| Thu Jan 17 16:59:31.179 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:31.660 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:31.660 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:31.660 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:31.723 [conn28] end connection 127.0.0.4:33681 (3 connections now open) m31001| Thu Jan 17 16:59:31.723 [initandlisten] connection accepted from 127.0.0.4:42669 #31 (4 connections now open) m31003| Thu Jan 17 16:59:32.261 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:59:32.261 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:32.269 [conn29] end connection 127.0.0.5:42471 (3 connections now open) m31001| Thu Jan 17 16:59:32.270 [initandlisten] connection accepted from 127.0.0.5:50589 #32 (4 connections now open) m31000| Thu Jan 17 16:59:33.152 [conn55] end connection 127.0.0.3:46639 (5 connections now open) m31001| Thu Jan 17 16:59:33.152 [rsSyncNotifier] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31001| Thu Jan 17 16:59:33.152 [rsSyncNotifier] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31001| Thu Jan 17 16:59:33.152 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 208, "optime" : { "t" : 1358459973000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:33Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358459968000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:59:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 207, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:33Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:33Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358459968000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T21:59:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:32Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:31Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:32Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:33Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459928000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:58:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:32Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:33Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:32Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 206, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31001| Thu Jan 17 16:59:34.152 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31001| Thu Jan 17 16:59:34.152 [rsSyncNotifier] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31001| Thu Jan 17 16:59:34.152 [rsSyncNotifier] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) m31000| Thu Jan 17 16:59:34.152 [initandlisten] connection accepted from 127.0.0.3:42948 #59 (6 connections now open) m31001| Thu Jan 17 16:59:38.203 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31002| Thu Jan 17 16:59:38.203 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 16:59:38.203 [initandlisten] connection accepted from 127.0.0.3:41447 #60 (7 connections now open) m31000| Thu Jan 17 16:59:38.203 [initandlisten] connection accepted from 127.0.0.4:47022 #61 (8 connections now open) m31001| Thu Jan 17 16:59:38.204 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 16:59:38.204 [conn59] end connection 127.0.0.3:42948 (7 connections now open) m31000| Thu Jan 17 16:59:38.204 [initandlisten] connection accepted from 127.0.0.3:42284 #62 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:38Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 213, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459968000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:28Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459973000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 212, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:37Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:38Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358459973000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:33Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:38Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 211, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:37Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:38Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 211, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:59:40.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 16:59:41.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:43Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 218, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 217, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:43Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:42Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:42Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:43Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358459978000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:38Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:42Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 216, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:59:45.059 [conn16] end connection 127.0.0.3:45019 (2 connections now open) m31003| Thu Jan 17 16:59:45.060 [initandlisten] connection accepted from 127.0.0.3:53925 #18 (3 connections now open) m31003| Thu Jan 17 16:59:45.726 [conn17] end connection 127.0.0.4:39442 (2 connections now open) m31003| Thu Jan 17 16:59:45.726 [initandlisten] connection accepted from 127.0.0.4:36182 #19 (3 connections now open) m31001| Thu Jan 17 16:59:46.206 [conn30] end connection 127.0.0.2:43899 (3 connections now open) m31001| Thu Jan 17 16:59:46.206 [initandlisten] connection accepted from 127.0.0.2:40333 #33 (4 connections now open) m31002| Thu Jan 17 16:59:46.272 [conn29] end connection 127.0.0.5:43299 (5 connections now open) m31002| Thu Jan 17 16:59:46.272 [initandlisten] connection accepted from 127.0.0.5:45669 #32 (6 connections now open) m31000| Thu Jan 17 16:59:47.060 [conn57] end connection 127.0.0.3:55993 (7 connections now open) m31000| Thu Jan 17 16:59:47.060 [initandlisten] connection accepted from 127.0.0.3:35107 #63 (8 connections now open) m31000| Thu Jan 17 16:59:47.726 [conn58] end connection 127.0.0.4:50545 (7 connections now open) m31000| Thu Jan 17 16:59:47.726 [initandlisten] connection accepted from 127.0.0.4:58213 #64 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:48Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 223, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 222, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:47Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:48Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:48Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:47Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:48Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:48Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358459983000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:43Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 221, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 16:59:48.388 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Thu Jan 17 16:59:48.388 [conn25] end connection 127.0.0.5:42969 (5 connections now open) m31001| Thu Jan 17 16:59:48.388 [initandlisten] connection accepted from 127.0.0.5:58822 #34 (5 connections now open) m31003| Thu Jan 17 16:59:48.389 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31002| Thu Jan 17 16:59:48.389 [conn26] end connection 127.0.0.5:33272 (4 connections now open) m31001| Thu Jan 17 16:59:48.389 [initandlisten] connection accepted from 127.0.0.5:59493 #35 (6 connections now open) m31001| Thu Jan 17 16:59:48.390 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Thu Jan 17 16:59:48.390 [initandlisten] connection accepted from 127.0.0.3:33306 #65 (9 connections now open) m31003| Thu Jan 17 16:59:50.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:53Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 228, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:41Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 24, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 227, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:53Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 24, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:52Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:51Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:52Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:53Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:52Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:52Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 226, "optime" : { "t" : 1358459988000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:48Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.3:31001 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 8 } } m31000| Thu Jan 17 16:59:53.309 [conn56] replSet replSetReconfig config object parses ok, 4 members specified m31000| Thu Jan 17 16:59:54.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:59:55.147 [conn56] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [5] server [127.0.0.5:31003] m31000| Thu Jan 17 16:59:55.147 [conn56] replSet replSetReconfig [2] m31000| Thu Jan 17 16:59:55.147 [conn56] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 16:59:55.148 [conn56] replSet saveConfigLocally done m31000| Thu Jan 17 16:59:55.148 [conn56] replSet relinquishing primary state m31000| Thu Jan 17 16:59:55.148 [conn56] replSet SECONDARY m31000| Thu Jan 17 16:59:55.148 [conn56] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:59:55.148 [conn56] replSet PRIMARY Thu Jan 17 16:59:55.148 DBClientCursor::init call() failed m31000| Thu Jan 17 16:59:55.148 [conn56] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:59:55.149 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31001| Thu Jan 17 16:59:55.149 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31001| Thu Jan 17 16:59:55.149 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 16:59:55.149 [conn56] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 40.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 0.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 8.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:983 reslen:71 1839ms m31000| Thu Jan 17 16:59:55.149 [conn62] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:42284] m31000| Thu Jan 17 16:59:55.149 [conn65] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:33306] m31000| Thu Jan 17 16:59:55.149 [conn60] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:41447] m31000| Thu Jan 17 16:59:55.149 [conn61] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:47022] m31000| Thu Jan 17 16:59:55.149 [conn56] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56506] m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 16:59:55.149 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Thu Jan 17 16:59:55.149 [rsMgr] replSet relinquishing primary state m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 16:59:55.149 [rsMgr] replSet SECONDARY m31000| Thu Jan 17 16:59:55.149 [rsMgr] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31000| Thu Jan 17 16:59:55.149 [rsMgr] replSet can't see a majority, will not try to elect self Thu Jan 17 16:59:55.149 trying reconnect to 127.0.0.2:31000 m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 16:59:55.149 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY Thu Jan 17 16:59:55.149 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 16:59:55.149 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' m31000| Thu Jan 17 16:59:55.149 [initandlisten] connection accepted from 127.0.0.1:56507 #66 (5 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:55Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 230, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 26, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 229, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 26, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:55Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:54Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T21:59:55Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 228, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31002| Thu Jan 17 16:59:55.728 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:59:55.728 [rsMgr] replset msgReceivedNewConfig version: version: 8 m31002| Thu Jan 17 16:59:55.728 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 16:59:55.737 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 16:59:55.737 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 16:59:55.737 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 16:59:55.738 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 16:59:55.738 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 16:59:55.738 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 16:59:55.738 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 16:59:55.738 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 16:59:56.208 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 16:59:56.208 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:59:56.275 [rsMgr] replset msgReceivedNewConfig version: version: 8 m31003| Thu Jan 17 16:59:56.275 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 16:59:56.286 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 16:59:56.286 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 16:59:56.286 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 16:59:56.286 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 16:59:56.286 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 16:59:56.286 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:59:57.062 [rsMgr] replset msgReceivedNewConfig version: version: 8 m31001| Thu Jan 17 16:59:57.062 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 16:59:57.062 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 16:59:57.076 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 16:59:57.076 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 16:59:57.076 [rsMgr] replset msgReceivedNewConfig version: version: 8 m31001| Thu Jan 17 16:59:57.076 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 8 8 m31001| Thu Jan 17 16:59:57.076 [rsMgr] replset msgReceivedNewConfig version: version: 8 m31001| Thu Jan 17 16:59:57.076 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 8 8 m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 16:59:57.076 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Thu Jan 17 16:59:57.147 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:57.728 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:57.728 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 16:59:57.728 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:59:58.274 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:59:58.275 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:59.062 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:59.062 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 16:59:59.062 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 16:59:59.738 [conn19] end connection 127.0.0.4:36182 (2 connections now open) m31003| Thu Jan 17 16:59:59.738 [initandlisten] connection accepted from 127.0.0.4:35008 #20 (3 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:00Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 235, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:59Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 234, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:00Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T21:59:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 233, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:00:00.267 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 17:00:00.267 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31003| Thu Jan 17 17:00:00.267 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:00:00.287 [conn32] end connection 127.0.0.5:45669 (3 connections now open) m31002| Thu Jan 17 17:00:00.287 [initandlisten] connection accepted from 127.0.0.5:39419 #33 (4 connections now open) m31003| Thu Jan 17 17:00:01.077 [conn18] end connection 127.0.0.3:53925 (2 connections now open) m31003| Thu Jan 17 17:00:01.077 [initandlisten] connection accepted from 127.0.0.3:47572 #21 (3 connections now open) m31001| Thu Jan 17 17:00:01.150 [conn33] end connection 127.0.0.2:40333 (5 connections now open) m31001| Thu Jan 17 17:00:01.150 [initandlisten] connection accepted from 127.0.0.2:46895 #36 (6 connections now open) m31000| Thu Jan 17 17:00:01.151 [rsMgr] replSet info electSelf 0 m31002| Thu Jan 17 17:00:01.151 [conn30] replSet info voting yea for 127.0.0.2:31000 (0) m31001| Thu Jan 17 17:00:01.151 [conn36] replSet info voting yea for 127.0.0.2:31000 (0) m31000| Thu Jan 17 17:00:01.304 [rsMgr] replSet PRIMARY m31000| Thu Jan 17 17:00:01.739 [conn64] end connection 127.0.0.4:58213 (4 connections now open) m31000| Thu Jan 17 17:00:01.739 [initandlisten] connection accepted from 127.0.0.4:52233 #67 (5 connections now open) m31002| Thu Jan 17 17:00:01.739 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31000| Thu Jan 17 17:00:03.077 [conn63] end connection 127.0.0.3:35107 (4 connections now open) m31000| Thu Jan 17 17:00:03.077 [initandlisten] connection accepted from 127.0.0.3:51459 #68 (5 connections now open) m31001| Thu Jan 17 17:00:03.078 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 17:00:05.149 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 17:00:05.149 [initandlisten] connection accepted from 127.0.0.3:52774 #69 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 240, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 8, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 239, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 10, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:05Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:04Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:05Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358459948000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:08Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 238, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31000| Thu Jan 17 17:00:07.147 [rsHealthPoll] replset info 127.0.0.5:31003 heartbeat failed, retrying m31002| Thu Jan 17 17:00:08.204 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:00:08.204 [conn51] end connection 127.0.0.4:41728 (5 connections now open) m31000| Thu Jan 17 17:00:08.205 [initandlisten] connection accepted from 127.0.0.4:34947 #70 (6 connections now open) m31002| Thu Jan 17 17:00:08.205 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 17:00:08.206 [initandlisten] connection accepted from 127.0.0.4:44146 #71 (7 connections now open) m31002| Thu Jan 17 17:00:08.207 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:00:08.207 [conn70] end connection 127.0.0.4:34947 (6 connections now open) m31000| Thu Jan 17 17:00:08.207 [initandlisten] connection accepted from 127.0.0.4:48971 #72 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 245, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 13, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:09Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 15, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:10Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460005000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 243, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:00:10.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:00:15.152 [conn30] end connection 127.0.0.2:52497 (3 connections now open) m31002| Thu Jan 17 17:00:15.153 [initandlisten] connection accepted from 127.0.0.2:55972 #34 (4 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 250, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 18, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 249, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 20, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:15Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:14Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:15Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460010000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 248, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31001| Thu Jan 17 17:00:15.741 [conn31] end connection 127.0.0.4:42669 (5 connections now open) m31001| Thu Jan 17 17:00:15.741 [initandlisten] connection accepted from 127.0.0.4:37760 #37 (6 connections now open) m31001| Thu Jan 17 17:00:16.290 [conn32] end connection 127.0.0.5:50589 (5 connections now open) m31001| Thu Jan 17 17:00:16.290 [initandlisten] connection accepted from 127.0.0.5:43547 #38 (6 connections now open) m31002| Thu Jan 17 17:00:17.080 [conn31] end connection 127.0.0.3:34926 (3 connections now open) m31002| Thu Jan 17 17:00:17.080 [initandlisten] connection accepted from 127.0.0.3:51126 #35 (4 connections now open) m31000| Thu Jan 17 17:00:20.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 255, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 23, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:19Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 254, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 25, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 25, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:20Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 253, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:00:20.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 260, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 28, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 259, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 28, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 28, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 30, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:25Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 30, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:24Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:25Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 29, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 258, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:00:29.744 [conn20] end connection 127.0.0.4:35008 (2 connections now open) m31003| Thu Jan 17 17:00:29.744 [initandlisten] connection accepted from 127.0.0.4:59233 #22 (3 connections now open) m31000| Thu Jan 17 17:00:30.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 265, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 33, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:29Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 264, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 33, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 35, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 263, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 35, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:30Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 34, "optime" : { "t" : 1358460025000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 263, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:00:30.267 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 17:00:30.267 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31002| Thu Jan 17 17:00:30.293 [conn33] end connection 127.0.0.5:39419 (3 connections now open) m31002| Thu Jan 17 17:00:30.293 [initandlisten] connection accepted from 127.0.0.5:33012 #36 (4 connections now open) m31003| Thu Jan 17 17:00:31.083 [conn21] end connection 127.0.0.3:47572 (2 connections now open) m31003| Thu Jan 17 17:00:31.083 [initandlisten] connection accepted from 127.0.0.3:50888 #23 (3 connections now open) m31001| Thu Jan 17 17:00:31.156 [conn36] end connection 127.0.0.2:46895 (5 connections now open) m31001| Thu Jan 17 17:00:31.156 [initandlisten] connection accepted from 127.0.0.2:52901 #39 (6 connections now open) m31000| Thu Jan 17 17:00:31.744 [conn67] end connection 127.0.0.4:52233 (6 connections now open) m31000| Thu Jan 17 17:00:31.745 [initandlisten] connection accepted from 127.0.0.4:41820 #73 (7 connections now open) m31000| Thu Jan 17 17:00:33.083 [conn68] end connection 127.0.0.3:51459 (6 connections now open) m31000| Thu Jan 17 17:00:33.084 [initandlisten] connection accepted from 127.0.0.3:53954 #74 (7 connections now open) m31003| Thu Jan 17 17:00:33.491 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31001| Thu Jan 17 17:00:33.491 [conn34] end connection 127.0.0.5:58822 (5 connections now open) m31002| Thu Jan 17 17:00:33.491 [initandlisten] connection accepted from 127.0.0.5:32857 #37 (5 connections now open) m31003| Thu Jan 17 17:00:33.492 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Thu Jan 17 17:00:33.492 [conn35] end connection 127.0.0.5:59493 (4 connections now open) m31002| Thu Jan 17 17:00:33.493 [initandlisten] connection accepted from 127.0.0.5:54781 #38 (6 connections now open) m31000| Thu Jan 17 17:00:33.494 [conn54] end connection 127.0.0.4:32972 (6 connections now open) m31002| Thu Jan 17 17:00:33.494 [rsGhostSync] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31002| Thu Jan 17 17:00:33.494 [rsGhostSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31002| Thu Jan 17 17:00:33.494 [rsGhostSync] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31002| Thu Jan 17 17:00:33.494 [rsGhostSync] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) m31001| Thu Jan 17 17:00:35.150 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:00:35.150 [initandlisten] connection accepted from 127.0.0.3:59581 #75 (7 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 270, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:07Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 38, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 269, "optime" : { "t" : 1358459995000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:55Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 38, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 40, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:35Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 268, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 40, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:34Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:00:35Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358459993000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T21:59:53Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 268, "optime" : { "t" : 1358460030000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:30Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.4:31002 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 9 } } m31000| Thu Jan 17 17:00:35.247 [conn66] replSet replSetReconfig config object parses ok, 4 members specified m31002| Thu Jan 17 17:00:35.498 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Thu Jan 17 17:00:35.498 [initandlisten] connection accepted from 127.0.0.4:44537 #76 (8 connections now open) m31003| Thu Jan 17 17:00:36.295 [rsMgr] replSet I don't see a primary and I can't elect myself m31003| Thu Jan 17 17:00:42.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:00:45.158 [conn34] end connection 127.0.0.2:55972 (5 connections now open) m31002| Thu Jan 17 17:00:45.159 [initandlisten] connection accepted from 127.0.0.2:56357 #39 (6 connections now open) m31001| Thu Jan 17 17:00:45.747 [conn37] end connection 127.0.0.4:37760 (3 connections now open) m31001| Thu Jan 17 17:00:45.747 [initandlisten] connection accepted from 127.0.0.4:56182 #40 (4 connections now open) m31001| Thu Jan 17 17:00:46.296 [conn38] end connection 127.0.0.5:43547 (3 connections now open) m31001| Thu Jan 17 17:00:46.297 [initandlisten] connection accepted from 127.0.0.5:40081 #41 (4 connections now open) m31002| Thu Jan 17 17:00:47.086 [conn35] end connection 127.0.0.3:51126 (5 connections now open) m31002| Thu Jan 17 17:00:47.086 [initandlisten] connection accepted from 127.0.0.3:50443 #40 (6 connections now open) m31000| Thu Jan 17 17:00:50.147 [conn66] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:00:52.267 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 17:00:59.750 [conn22] end connection 127.0.0.4:59233 (2 connections now open) m31003| Thu Jan 17 17:00:59.750 [initandlisten] connection accepted from 127.0.0.4:48277 #24 (3 connections now open) m31000| Thu Jan 17 17:01:00.147 [conn66] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [6] server [127.0.0.5:31003] m31000| Thu Jan 17 17:01:00.147 [conn66] replSet replSetReconfig [2] m31000| Thu Jan 17 17:01:00.147 [conn66] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 17:01:00.166 [conn66] replSet saveConfigLocally done m31000| Thu Jan 17 17:01:00.166 [conn66] replSet relinquishing primary state m31000| Thu Jan 17 17:01:00.166 [conn66] replSet SECONDARY m31000| Thu Jan 17 17:01:00.166 [conn66] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 17:01:00.166 [conn66] replSet PRIMARY Thu Jan 17 17:01:00.166 DBClientCursor::init call() failed m31000| Thu Jan 17 17:01:00.166 [conn66] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 17:01:00.166 [conn69] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:52774] m31000| Thu Jan 17 17:01:00.166 [conn66] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 0.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 40.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 9.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:18434 reslen:71 24918ms m31000| Thu Jan 17 17:01:00.166 [conn71] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:44146] m31000| Thu Jan 17 17:01:00.166 [conn66] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56507] Caught exception error doing query: failed, this is normal after reconfig. m31002| Thu Jan 17 17:01:00.166 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 17:01:00.166 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 17:01:00.166 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 17:01:00.166 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 17:01:00.166 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 17:01:00.166 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY Thu Jan 17 17:01:00.166 trying reconnect to 127.0.0.2:31000 Thu Jan 17 17:01:00.167 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 17:01:00.167 [initandlisten] connection accepted from 127.0.0.1:56508 #77 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:00Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 295, "optime" : { "t" : 1358460060000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:01:00Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 63, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 294, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 63, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 63, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 65, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 65, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 293, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 65, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:00Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 64, "optime" : { "t" : 1358460015000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 64, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:00:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:00:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 293, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31002| Thu Jan 17 17:01:00.299 [conn36] end connection 127.0.0.5:33012 (5 connections now open) m31002| Thu Jan 17 17:01:00.299 [initandlisten] connection accepted from 127.0.0.5:48940 #41 (6 connections now open) m31001| Thu Jan 17 17:01:01.089 [rsMgr] replset msgReceivedNewConfig version: version: 9 m31003| Thu Jan 17 17:01:01.089 [conn23] end connection 127.0.0.3:50888 (2 connections now open) m31003| Thu Jan 17 17:01:01.089 [initandlisten] connection accepted from 127.0.0.3:55277 #25 (3 connections now open) m31001| Thu Jan 17 17:01:01.089 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 17:01:01.119 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 17:01:01.119 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 17:01:01.119 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 17:01:01.119 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 17:01:01.119 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 17:01:01.120 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 17:01:01.120 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 17:01:01.120 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31000| Thu Jan 17 17:01:01.162 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:01:01.162 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:01:01.750 [conn73] end connection 127.0.0.4:41820 (5 connections now open) m31000| Thu Jan 17 17:01:01.750 [initandlisten] connection accepted from 127.0.0.4:44166 #78 (6 connections now open) m31002| Thu Jan 17 17:01:01.750 [rsMgr] replset msgReceivedNewConfig version: version: 9 m31002| Thu Jan 17 17:01:01.750 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 17:01:01.751 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 17:01:01.752 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 17:01:01.752 [rsMgr] replset msgReceivedNewConfig version: version: 9 m31002| Thu Jan 17 17:01:01.752 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 9 9 m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 17:01:01.752 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31003| Thu Jan 17 17:01:02.300 [rsMgr] replset msgReceivedNewConfig version: version: 9 m31003| Thu Jan 17 17:01:02.300 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 17:01:02.309 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 17:01:02.309 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 17:01:02.309 [rsMgr] replset msgReceivedNewConfig version: version: 9 m31003| Thu Jan 17 17:01:02.310 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 9 9 m31003| Thu Jan 17 17:01:02.310 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 17:01:02.310 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 17:01:02.310 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 17:01:02.310 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 17:01:03.089 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:01:03.089 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:01:03.089 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:01:03.750 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:01:03.750 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:01:03.751 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:01:04.267 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:01:04.300 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:01:04.300 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:01:05.154 [conn75] end connection 127.0.0.3:59581 (5 connections now open) m31001| Thu Jan 17 17:01:05.154 [rsSyncNotifier] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31001| Thu Jan 17 17:01:05.154 [rsSyncNotifier] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31001| Thu Jan 17 17:01:05.154 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:05Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 300, "optime" : { "t" : 1358460065000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:05Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358460060000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:01:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 299, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:05Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:05Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358460060000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:01:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:04Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 298, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:03Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:04Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:05Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460020000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:04Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:05Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:04Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 298, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31001| Thu Jan 17 17:01:06.154 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31001| Thu Jan 17 17:01:06.154 [rsSyncNotifier] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31001| Thu Jan 17 17:01:06.154 [rsSyncNotifier] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) m31000| Thu Jan 17 17:01:06.155 [initandlisten] connection accepted from 127.0.0.3:38432 #79 (6 connections now open) m31002| Thu Jan 17 17:01:08.167 [conn39] end connection 127.0.0.2:56357 (5 connections now open) m31002| Thu Jan 17 17:01:08.168 [initandlisten] connection accepted from 127.0.0.2:38447 #42 (6 connections now open) m31002| Thu Jan 17 17:01:09.121 [conn40] end connection 127.0.0.3:50443 (5 connections now open) m31002| Thu Jan 17 17:01:09.121 [initandlisten] connection accepted from 127.0.0.3:35087 #43 (6 connections now open) m31001| Thu Jan 17 17:01:09.753 [conn40] end connection 127.0.0.4:56182 (3 connections now open) m31001| Thu Jan 17 17:01:09.753 [initandlisten] connection accepted from 127.0.0.4:34460 #42 (4 connections now open) m31000| Thu Jan 17 17:01:10.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Thu Jan 17 17:01:10.166 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31001| Thu Jan 17 17:01:10.166 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 17:01:10.166 [initandlisten] connection accepted from 127.0.0.4:54237 #80 (7 connections now open) m31000| Thu Jan 17 17:01:10.166 [initandlisten] connection accepted from 127.0.0.3:53340 #81 (8 connections now open) m31001| Thu Jan 17 17:01:10.168 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:01:10.168 [conn79] end connection 127.0.0.3:38432 (7 connections now open) m31000| Thu Jan 17 17:01:10.168 [initandlisten] connection accepted from 127.0.0.3:40604 #82 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:10Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 305, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460060000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358460065000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 304, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:10Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358460065000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:05Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:10Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 303, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:09Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:10Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:08Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:09Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 303, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31001| Thu Jan 17 17:01:10.311 [conn41] end connection 127.0.0.5:40081 (3 connections now open) m31001| Thu Jan 17 17:01:10.311 [initandlisten] connection accepted from 127.0.0.5:56989 #43 (4 connections now open) m31003| Thu Jan 17 17:01:12.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:15Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 310, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 309, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:15Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:14Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 308, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:13Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:14Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:15Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460070000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:10Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:14Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:15Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:14Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 308, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 17:01:15.587 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Thu Jan 17 17:01:15.587 [conn37] end connection 127.0.0.5:32857 (5 connections now open) m31001| Thu Jan 17 17:01:15.588 [initandlisten] connection accepted from 127.0.0.5:58275 #44 (5 connections now open) m31003| Thu Jan 17 17:01:15.589 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31002| Thu Jan 17 17:01:15.589 [conn38] end connection 127.0.0.5:54781 (4 connections now open) m31001| Thu Jan 17 17:01:15.589 [initandlisten] connection accepted from 127.0.0.5:33733 #45 (6 connections now open) m31000| Thu Jan 17 17:01:15.590 [initandlisten] connection accepted from 127.0.0.3:41562 #83 (9 connections now open) m31001| Thu Jan 17 17:01:15.590 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31000| Thu Jan 17 17:01:20.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 315, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 314, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 19, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 313, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:01:20Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358460035000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:00:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:01:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:01:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 313, "optime" : { "t" : 1358460075000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:15Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.3:31001 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 10 } } m31000| Thu Jan 17 17:01:20.216 [conn77] replSet replSetReconfig config object parses ok, 4 members specified m31003| Thu Jan 17 17:01:22.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 17:01:23.124 [conn25] end connection 127.0.0.3:55277 (2 connections now open) m31003| Thu Jan 17 17:01:23.124 [initandlisten] connection accepted from 127.0.0.3:39119 #26 (3 connections now open) m31003| Thu Jan 17 17:01:23.756 [conn24] end connection 127.0.0.4:48277 (2 connections now open) m31003| Thu Jan 17 17:01:23.756 [initandlisten] connection accepted from 127.0.0.4:37024 #27 (3 connections now open) m31001| Thu Jan 17 17:01:24.170 [conn39] end connection 127.0.0.2:52901 (5 connections now open) m31001| Thu Jan 17 17:01:24.171 [initandlisten] connection accepted from 127.0.0.2:45616 #46 (6 connections now open) m31002| Thu Jan 17 17:01:24.314 [conn41] end connection 127.0.0.5:48940 (3 connections now open) m31002| Thu Jan 17 17:01:24.314 [initandlisten] connection accepted from 127.0.0.5:35255 #44 (4 connections now open) m31000| Thu Jan 17 17:01:25.124 [conn74] end connection 127.0.0.3:53954 (8 connections now open) m31000| Thu Jan 17 17:01:25.124 [initandlisten] connection accepted from 127.0.0.3:46256 #84 (9 connections now open) m31000| Thu Jan 17 17:01:25.756 [conn78] end connection 127.0.0.4:44166 (8 connections now open) m31000| Thu Jan 17 17:01:25.756 [initandlisten] connection accepted from 127.0.0.4:33595 #85 (9 connections now open) m31000| Thu Jan 17 17:01:30.147 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:01:32.307 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 17:01:32.307 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31002| Thu Jan 17 17:01:38.173 [conn42] end connection 127.0.0.2:38447 (3 connections now open) m31002| Thu Jan 17 17:01:38.173 [initandlisten] connection accepted from 127.0.0.2:58375 #45 (4 connections now open) m31002| Thu Jan 17 17:01:39.126 [conn43] end connection 127.0.0.3:35087 (3 connections now open) m31002| Thu Jan 17 17:01:39.127 [initandlisten] connection accepted from 127.0.0.3:48306 #46 (4 connections now open) m31001| Thu Jan 17 17:01:39.759 [conn42] end connection 127.0.0.4:34460 (5 connections now open) m31001| Thu Jan 17 17:01:39.759 [initandlisten] connection accepted from 127.0.0.4:59826 #47 (6 connections now open) m31000| Thu Jan 17 17:01:40.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Thu Jan 17 17:01:40.167 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:01:40.167 [conn72] end connection 127.0.0.4:48971 (8 connections now open) m31000| Thu Jan 17 17:01:40.168 [initandlisten] connection accepted from 127.0.0.4:43290 #86 (9 connections now open) m31001| Thu Jan 17 17:01:40.317 [conn43] end connection 127.0.0.5:56989 (5 connections now open) m31001| Thu Jan 17 17:01:40.317 [initandlisten] connection accepted from 127.0.0.5:60001 #48 (6 connections now open) m31003| Thu Jan 17 17:01:44.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 17:01:50.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:01:53.129 [conn26] end connection 127.0.0.3:39119 (2 connections now open) m31003| Thu Jan 17 17:01:53.129 [initandlisten] connection accepted from 127.0.0.3:41340 #28 (3 connections now open) m31003| Thu Jan 17 17:01:53.761 [conn27] end connection 127.0.0.4:37024 (2 connections now open) m31003| Thu Jan 17 17:01:53.762 [initandlisten] connection accepted from 127.0.0.4:40427 #29 (3 connections now open) m31001| Thu Jan 17 17:01:54.177 [conn46] end connection 127.0.0.2:45616 (5 connections now open) m31001| Thu Jan 17 17:01:54.177 [initandlisten] connection accepted from 127.0.0.2:49011 #49 (6 connections now open) m31003| Thu Jan 17 17:01:54.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:01:54.319 [conn44] end connection 127.0.0.5:35255 (3 connections now open) m31002| Thu Jan 17 17:01:54.319 [initandlisten] connection accepted from 127.0.0.5:48631 #47 (4 connections now open) m31000| Thu Jan 17 17:01:55.130 [conn84] end connection 127.0.0.3:46256 (8 connections now open) m31000| Thu Jan 17 17:01:55.130 [initandlisten] connection accepted from 127.0.0.3:37663 #87 (10 connections now open) m31000| Thu Jan 17 17:01:55.762 [conn85] end connection 127.0.0.4:33595 (8 connections now open) m31000| Thu Jan 17 17:01:55.762 [initandlisten] connection accepted from 127.0.0.4:39393 #88 (9 connections now open) m31003| Thu Jan 17 17:01:56.320 [rsMgr] replSet I don't see a primary and I can't elect myself m31000| Thu Jan 17 17:02:00.147 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 17:02:00.147 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31002| Thu Jan 17 17:02:08.179 [conn45] end connection 127.0.0.2:58375 (3 connections now open) m31002| Thu Jan 17 17:02:08.179 [initandlisten] connection accepted from 127.0.0.2:51834 #48 (4 connections now open) m31002| Thu Jan 17 17:02:09.132 [conn46] end connection 127.0.0.3:48306 (3 connections now open) m31002| Thu Jan 17 17:02:09.132 [initandlisten] connection accepted from 127.0.0.3:57659 #49 (4 connections now open) m31001| Thu Jan 17 17:02:09.764 [conn47] end connection 127.0.0.4:59826 (5 connections now open) m31001| Thu Jan 17 17:02:09.765 [initandlisten] connection accepted from 127.0.0.4:41540 #50 (6 connections now open) m31000| Thu Jan 17 17:02:10.147 [conn77] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Thu Jan 17 17:02:10.322 [conn48] end connection 127.0.0.5:60001 (5 connections now open) m31001| Thu Jan 17 17:02:10.322 [initandlisten] connection accepted from 127.0.0.5:46955 #51 (6 connections now open) m31003| Thu Jan 17 17:02:16.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 17:02:20.147 [conn77] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [6] server [127.0.0.5:31003] m31000| Thu Jan 17 17:02:20.147 [conn77] replSet replSetReconfig [2] m31000| Thu Jan 17 17:02:20.147 [conn77] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 17:02:20.159 [conn77] replSet saveConfigLocally done m31000| Thu Jan 17 17:02:20.159 [conn77] replSet relinquishing primary state m31000| Thu Jan 17 17:02:20.159 [conn77] replSet SECONDARY m31000| Thu Jan 17 17:02:20.159 [conn77] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 17:02:20.159 [conn77] replSet PRIMARY Thu Jan 17 17:02:20.159 DBClientCursor::init call() failed m31000| Thu Jan 17 17:02:20.159 [conn77] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 17:02:20.159 [conn77] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 40.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 0.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 10.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:12050 reslen:71 59943ms m31000| Thu Jan 17 17:02:20.159 [conn81] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:53340] m31002| Thu Jan 17 17:02:20.159 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31001| Thu Jan 17 17:02:20.159 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 17:02:20.160 [conn77] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56508] m31000| Thu Jan 17 17:02:20.159 [conn80] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:54237] m31000| Thu Jan 17 17:02:20.160 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 17:02:20.160 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 17:02:20.160 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31000| Thu Jan 17 17:02:20.160 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY Thu Jan 17 17:02:20.160 trying reconnect to 127.0.0.2:31000 m31000| Thu Jan 17 17:02:20.160 [initandlisten] connection accepted from 127.0.0.1:56509 #89 (7 connections now open) Thu Jan 17 17:02:20.160 reconnect 127.0.0.2:31000 ok { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:20Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 375, "optime" : { "t" : 1358460140000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:02:20Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:20Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 374, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:20Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:20Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 373, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:19Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:20Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:03Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:18Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 373, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31000| Thu Jan 17 17:02:20.181 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:02:20.181 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:02:21.135 [rsMgr] replset msgReceivedNewConfig version: version: 10 m31001| Thu Jan 17 17:02:21.135 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 17:02:21.142 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 17:02:21.142 [rsMgr] replSet replSetReconfig new config saved locally m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31001| Thu Jan 17 17:02:21.143 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31002| Thu Jan 17 17:02:21.767 [rsMgr] replset msgReceivedNewConfig version: version: 10 m31002| Thu Jan 17 17:02:21.767 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 17:02:21.768 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 17:02:21.768 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 17:02:21.768 [rsMgr] replset msgReceivedNewConfig version: version: 10 m31002| Thu Jan 17 17:02:21.768 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 10 10 m31002| Thu Jan 17 17:02:21.768 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 17:02:21.768 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 17:02:21.769 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 17:02:21.769 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 17:02:21.769 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state PRIMARY m31002| Thu Jan 17 17:02:21.769 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31003| Thu Jan 17 17:02:22.325 [rsMgr] replset msgReceivedNewConfig version: version: 10 m31003| Thu Jan 17 17:02:22.325 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 17:02:22.326 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 17:02:22.326 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 17:02:22.326 [rsMgr] replset msgReceivedNewConfig version: version: 10 m31003| Thu Jan 17 17:02:22.326 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 10 10 m31003| Thu Jan 17 17:02:22.326 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 17:02:22.326 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 17:02:22.326 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 17:02:22.326 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 17:02:23.135 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:02:23.135 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:02:23.135 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:02:23.767 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:02:23.767 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:02:23.767 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:02:24.325 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:02:24.325 [rsHealthPoll] Client::shutdown not called: rsHealthPoll { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 380, "optime" : { "t" : 1358460145000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:25Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358460140000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:02:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 379, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:25Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:25Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 4, "optime" : { "t" : 1358460140000, "i" : 2 }, "optimeDate" : ISODate("2013-01-17T22:02:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:24Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 378, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:23Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:24Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:25Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:24Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:24Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 378, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:02:26.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 17:02:30.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Thu Jan 17 17:02:30.160 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31001| Thu Jan 17 17:02:30.160 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 17:02:30.160 [initandlisten] connection accepted from 127.0.0.4:49860 #90 (8 connections now open) m31000| Thu Jan 17 17:02:30.160 [initandlisten] connection accepted from 127.0.0.3:53670 #91 (9 connections now open) m31002| Thu Jan 17 17:02:30.161 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:02:30.161 [conn86] end connection 127.0.0.4:43290 (8 connections now open) m31000| Thu Jan 17 17:02:30.161 [initandlisten] connection accepted from 127.0.0.4:38240 #92 (9 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:30Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 385, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460140000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358460145000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 384, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:30Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 9, "optime" : { "t" : 1358460145000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:25Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:30Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 383, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:29Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:30Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:28Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:29Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 383, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.4:31002 m31003| Thu Jan 17 17:02:30.759 [rsBackgroundSync] replSet syncing to: 127.0.0.4:31002 m31001| Thu Jan 17 17:02:30.759 [conn44] end connection 127.0.0.5:58275 (5 connections now open) m31002| Thu Jan 17 17:02:30.759 [initandlisten] connection accepted from 127.0.0.5:36592 #50 (5 connections now open) m31003| Thu Jan 17 17:02:30.760 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.4:31002 m31001| Thu Jan 17 17:02:30.760 [conn45] end connection 127.0.0.5:33733 (4 connections now open) m31002| Thu Jan 17 17:02:30.760 [initandlisten] connection accepted from 127.0.0.5:49384 #51 (6 connections now open) m31000| Thu Jan 17 17:02:30.761 [conn76] end connection 127.0.0.4:44537 (8 connections now open) m31002| Thu Jan 17 17:02:30.761 [rsGhostSync] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31002| Thu Jan 17 17:02:30.761 [rsGhostSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31002| Thu Jan 17 17:02:30.761 [rsGhostSync] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31002| Thu Jan 17 17:02:30.761 [rsGhostSync] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) m31003| Thu Jan 17 17:02:31.145 [conn28] end connection 127.0.0.3:41340 (2 connections now open) m31003| Thu Jan 17 17:02:31.145 [initandlisten] connection accepted from 127.0.0.3:55857 #30 (3 connections now open) m31003| Thu Jan 17 17:02:31.770 [conn29] end connection 127.0.0.4:40427 (2 connections now open) m31003| Thu Jan 17 17:02:31.770 [initandlisten] connection accepted from 127.0.0.4:45466 #31 (3 connections now open) m31001| Thu Jan 17 17:02:32.162 [conn49] end connection 127.0.0.2:49011 (3 connections now open) m31001| Thu Jan 17 17:02:32.162 [initandlisten] connection accepted from 127.0.0.2:59001 #52 (4 connections now open) m31002| Thu Jan 17 17:02:32.328 [conn47] end connection 127.0.0.5:48631 (5 connections now open) m31002| Thu Jan 17 17:02:32.328 [initandlisten] connection accepted from 127.0.0.5:40096 #52 (6 connections now open) m31000| Thu Jan 17 17:02:33.145 [conn87] end connection 127.0.0.3:37663 (7 connections now open) m31000| Thu Jan 17 17:02:33.145 [initandlisten] connection accepted from 127.0.0.3:33955 #93 (8 connections now open) m31000| Thu Jan 17 17:02:33.771 [conn88] end connection 127.0.0.4:39393 (7 connections now open) m31000| Thu Jan 17 17:02:33.771 [initandlisten] connection accepted from 127.0.0.4:59757 #94 (8 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:35Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 390, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 389, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:35Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 14, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:34Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 388, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:33Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:34Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:02:35Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460080000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:01:20Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:34Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:02:35Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "lastHeartbeat" : ISODate("2013-01-17T22:02:34Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 388, "optime" : { "t" : 1358460150000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:30Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.4:31002 { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ] } Reconfiguring replica set... { "replSetReconfig" : { "_id" : "testReplSet", "members" : [ { "_id" : 0, "host" : "127.0.0.2:31000", "priority" : 1 }, { "_id" : 1, "host" : "127.0.0.3:31001", "priority" : 0, "slaveDelay" : 0 }, { "_id" : 2, "host" : "127.0.0.4:31002", "priority" : 0, "slaveDelay" : 40 }, { "_id" : 3, "host" : "127.0.0.5:31003", "priority" : 0 } ], "version" : 11 } } m31000| Thu Jan 17 17:02:35.202 [conn89] replSet replSetReconfig config object parses ok, 4 members specified m31000| Thu Jan 17 17:02:35.772 [initandlisten] connection accepted from 127.0.0.4:40547 #95 (9 connections now open) m31002| Thu Jan 17 17:02:35.772 [rsGhostSync] handshake between 3 and 127.0.0.2:31000 m31003| Thu Jan 17 17:02:36.307 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:02:40.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Thu Jan 17 17:02:46.164 [conn48] end connection 127.0.0.2:51834 (5 connections now open) m31002| Thu Jan 17 17:02:46.164 [initandlisten] connection accepted from 127.0.0.2:40950 #53 (6 connections now open) m31003| Thu Jan 17 17:02:46.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:02:47.148 [conn49] end connection 127.0.0.3:57659 (5 connections now open) m31002| Thu Jan 17 17:02:47.148 [initandlisten] connection accepted from 127.0.0.3:36149 #54 (6 connections now open) m31001| Thu Jan 17 17:02:47.773 [conn50] end connection 127.0.0.4:41540 (3 connections now open) m31001| Thu Jan 17 17:02:47.773 [initandlisten] connection accepted from 127.0.0.4:56131 #53 (4 connections now open) m31001| Thu Jan 17 17:02:48.331 [conn51] end connection 127.0.0.5:46955 (3 connections now open) m31001| Thu Jan 17 17:02:48.331 [initandlisten] connection accepted from 127.0.0.5:33170 #54 (4 connections now open) m31000| Thu Jan 17 17:02:50.147 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:02:56.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 17:03:00.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Thu Jan 17 17:03:00.161 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:03:00.161 [conn82] end connection 127.0.0.3:40604 (8 connections now open) m31000| Thu Jan 17 17:03:00.161 [initandlisten] connection accepted from 127.0.0.3:48594 #96 (9 connections now open) m31003| Thu Jan 17 17:03:01.150 [conn30] end connection 127.0.0.3:55857 (2 connections now open) m31003| Thu Jan 17 17:03:01.151 [initandlisten] connection accepted from 127.0.0.3:51265 #32 (3 connections now open) m31003| Thu Jan 17 17:03:01.775 [conn31] end connection 127.0.0.4:45466 (2 connections now open) m31003| Thu Jan 17 17:03:01.776 [initandlisten] connection accepted from 127.0.0.4:43615 #33 (3 connections now open) m31001| Thu Jan 17 17:03:02.167 [conn52] end connection 127.0.0.2:59001 (3 connections now open) m31001| Thu Jan 17 17:03:02.168 [initandlisten] connection accepted from 127.0.0.2:59213 #55 (4 connections now open) m31002| Thu Jan 17 17:03:02.334 [conn52] end connection 127.0.0.5:40096 (5 connections now open) m31002| Thu Jan 17 17:03:02.334 [initandlisten] connection accepted from 127.0.0.5:48353 #55 (6 connections now open) m31000| Thu Jan 17 17:03:03.150 [conn93] end connection 127.0.0.3:33955 (8 connections now open) m31000| Thu Jan 17 17:03:03.151 [initandlisten] connection accepted from 127.0.0.3:52770 #97 (9 connections now open) m31000| Thu Jan 17 17:03:03.776 [conn94] end connection 127.0.0.4:59757 (8 connections now open) m31000| Thu Jan 17 17:03:03.776 [initandlisten] connection accepted from 127.0.0.4:51115 #98 (9 connections now open) m31003| Thu Jan 17 17:03:06.307 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 17:03:06.307 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31000| Thu Jan 17 17:03:10.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:03:12.336 [rsMgr] replSet I don't see a primary and I can't elect myself m31002| Thu Jan 17 17:03:16.170 [conn53] end connection 127.0.0.2:40950 (5 connections now open) m31002| Thu Jan 17 17:03:16.170 [initandlisten] connection accepted from 127.0.0.2:56557 #56 (6 connections now open) m31002| Thu Jan 17 17:03:17.153 [conn54] end connection 127.0.0.3:36149 (5 connections now open) m31002| Thu Jan 17 17:03:17.153 [initandlisten] connection accepted from 127.0.0.3:33181 #57 (6 connections now open) m31001| Thu Jan 17 17:03:17.779 [conn53] end connection 127.0.0.4:56131 (3 connections now open) m31001| Thu Jan 17 17:03:17.779 [initandlisten] connection accepted from 127.0.0.4:40341 #56 (4 connections now open) m31003| Thu Jan 17 17:03:18.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Thu Jan 17 17:03:18.337 [conn54] end connection 127.0.0.5:33170 (3 connections now open) m31001| Thu Jan 17 17:03:18.337 [initandlisten] connection accepted from 127.0.0.5:47555 #57 (5 connections now open) m31000| Thu Jan 17 17:03:20.147 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 17:03:20.147 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31003| Thu Jan 17 17:03:28.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31000| Thu Jan 17 17:03:30.147 [conn89] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:03:31.156 [conn32] end connection 127.0.0.3:51265 (2 connections now open) m31003| Thu Jan 17 17:03:31.156 [initandlisten] connection accepted from 127.0.0.3:36180 #34 (3 connections now open) m31003| Thu Jan 17 17:03:31.781 [conn33] end connection 127.0.0.4:43615 (2 connections now open) m31003| Thu Jan 17 17:03:31.781 [initandlisten] connection accepted from 127.0.0.4:36786 #35 (3 connections now open) m31001| Thu Jan 17 17:03:32.173 [conn55] end connection 127.0.0.2:59213 (3 connections now open) m31001| Thu Jan 17 17:03:32.174 [initandlisten] connection accepted from 127.0.0.2:60463 #58 (4 connections now open) m31002| Thu Jan 17 17:03:32.340 [conn55] end connection 127.0.0.5:48353 (5 connections now open) m31002| Thu Jan 17 17:03:32.340 [initandlisten] connection accepted from 127.0.0.5:58462 #58 (6 connections now open) m31000| Thu Jan 17 17:03:33.156 [conn97] end connection 127.0.0.3:52770 (8 connections now open) m31000| Thu Jan 17 17:03:33.156 [initandlisten] connection accepted from 127.0.0.3:55456 #99 (9 connections now open) m31000| Thu Jan 17 17:03:33.782 [conn98] end connection 127.0.0.4:51115 (8 connections now open) m31000| Thu Jan 17 17:03:33.782 [initandlisten] connection accepted from 127.0.0.4:36393 #100 (9 connections now open) m31000| Thu Jan 17 17:03:40.147 [conn89] replSet cmufcc requestHeartbeat 127.0.0.5:31003 : 9001 socket exception [6] server [127.0.0.5:31003] m31000| Thu Jan 17 17:03:40.147 [conn89] replSet replSetReconfig [2] m31000| Thu Jan 17 17:03:40.147 [conn89] replSet info saving a newer config version to local.system.replset m31000| Thu Jan 17 17:03:40.161 [conn89] replSet saveConfigLocally done m31000| Thu Jan 17 17:03:40.161 [conn89] replSet relinquishing primary state m31000| Thu Jan 17 17:03:40.161 [conn89] replSet SECONDARY m31000| Thu Jan 17 17:03:40.161 [conn89] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 17:03:40.161 [conn89] replSet PRIMARY Thu Jan 17 17:03:40.161 DBClientCursor::init call() failed m31002| Thu Jan 17 17:03:40.161 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 Caught exception error doing query: failed, this is normal after reconfig. m31001| Thu Jan 17 17:03:40.161 [conn58] end connection 127.0.0.2:60463 (3 connections now open) m31001| Thu Jan 17 17:03:40.161 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 17:03:40.161 [conn89] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 17:03:40.161 [conn90] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.4:49860] m31000| Thu Jan 17 17:03:40.161 [conn91] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.3:53670] m31000| Thu Jan 17 17:03:40.161 [conn89] command admin.$cmd command: { replSetReconfig: { _id: "testReplSet", members: [ { _id: 0.0, host: "127.0.0.2:31000", priority: 1.0 }, { _id: 1.0, host: "127.0.0.3:31001", priority: 0.0, slaveDelay: 0.0 }, { _id: 2.0, host: "127.0.0.4:31002", priority: 0.0, slaveDelay: 40.0 }, { _id: 3.0, host: "127.0.0.5:31003", priority: 0.0 } ], version: 11.0 } } ntoreturn:1 keyUpdates:0 locks(micros) W:13963 reslen:71 64959ms m31000| Thu Jan 17 17:03:40.161 [conn89] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:56509] m31000| Thu Jan 17 17:03:40.161 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31000| Thu Jan 17 17:03:40.161 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31000| Thu Jan 17 17:03:40.162 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Thu Jan 17 17:03:40.162 [rsMgr] replSet relinquishing primary state m31000| Thu Jan 17 17:03:40.162 [rsMgr] replSet SECONDARY m31000| Thu Jan 17 17:03:40.162 [rsMgr] replSet closing client sockets after relinquishing primary m31000| Thu Jan 17 17:03:40.162 [rsHealthPoll] couldn't connect to 127.0.0.3:31001: couldn't connect to server 127.0.0.3:31001 Thu Jan 17 17:03:40.162 trying reconnect to 127.0.0.2:31000 m31001| Thu Jan 17 17:03:40.162 [initandlisten] connection accepted from 127.0.0.2:45995 #59 (4 connections now open) m31001| Thu Jan 17 17:03:40.162 [conn59] end connection 127.0.0.2:45995 (3 connections now open) m31001| Thu Jan 17 17:03:40.162 [initandlisten] connection accepted from 127.0.0.2:48626 #60 (5 connections now open) Thu Jan 17 17:03:40.162 reconnect 127.0.0.2:31000 ok m31000| Thu Jan 17 17:03:40.162 [initandlisten] connection accepted from 127.0.0.1:56510 #101 (7 connections now open) m31000| Thu Jan 17 17:03:40.162 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31000| Thu Jan 17 17:03:40.162 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:40Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 455, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 454, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:39Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:40Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:40Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 453, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 79, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:39Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:40Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:38Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:39Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 78, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:38Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:39Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 453, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31000| Thu Jan 17 17:03:40.175 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31000| Thu Jan 17 17:03:40.175 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:03:41.158 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31001| Thu Jan 17 17:03:41.158 [rsMgr] replset msgReceivedNewConfig version: version: 11 m31001| Thu Jan 17 17:03:41.158 [rsMgr] replSet info saving a newer config version to local.system.replset m31001| Thu Jan 17 17:03:41.159 [rsMgr] replSet saveConfigLocally done m31001| Thu Jan 17 17:03:41.159 [rsMgr] replSet replSetReconfig new config saved locally m31000| Thu Jan 17 17:03:41.160 [conn99] end connection 127.0.0.3:55456 (6 connections now open) m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31000| Thu Jan 17 17:03:41.160 [initandlisten] connection accepted from 127.0.0.3:57462 #102 (7 connections now open) m31001| Thu Jan 17 17:03:41.160 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31001| Thu Jan 17 17:03:41.160 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 17:03:41.783 [rsMgr] replset msgReceivedNewConfig version: version: 11 m31002| Thu Jan 17 17:03:41.783 [rsMgr] replSet info saving a newer config version to local.system.replset m31002| Thu Jan 17 17:03:41.783 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31002| Thu Jan 17 17:03:41.784 [rsMgr] replSet saveConfigLocally done m31002| Thu Jan 17 17:03:41.784 [rsMgr] replSet replSetReconfig new config saved locally m31002| Thu Jan 17 17:03:41.784 [rsMgr] replset msgReceivedNewConfig version: version: 11 m31002| Thu Jan 17 17:03:41.784 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 11 11 m31000| Thu Jan 17 17:03:41.785 [conn100] end connection 127.0.0.4:36393 (6 connections now open) m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.5:31003 is up m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state SECONDARY m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31002| Thu Jan 17 17:03:41.785 [rsMgr] replSet I don't see a primary and I can't elect myself m31000| Thu Jan 17 17:03:41.785 [initandlisten] connection accepted from 127.0.0.4:44306 #103 (7 connections now open) m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.2:31000 is up m31002| Thu Jan 17 17:03:41.785 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state SECONDARY m31003| Thu Jan 17 17:03:42.342 [rsMgr] replset msgReceivedNewConfig version: version: 11 m31003| Thu Jan 17 17:03:42.342 [rsMgr] replSet info saving a newer config version to local.system.replset m31003| Thu Jan 17 17:03:42.343 [rsMgr] replSet saveConfigLocally done m31003| Thu Jan 17 17:03:42.343 [rsMgr] replSet replSetReconfig new config saved locally m31003| Thu Jan 17 17:03:42.343 [rsMgr] replset msgReceivedNewConfig version: version: 11 m31003| Thu Jan 17 17:03:42.343 [rsMgr] replSet info msgReceivedNewConfig but version isn't higher 11 11 m31003| Thu Jan 17 17:03:42.343 [rsHealthPoll] replSet member 127.0.0.4:31002 is up m31003| Thu Jan 17 17:03:42.343 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state SECONDARY m31003| Thu Jan 17 17:03:42.343 [rsHealthPoll] replSet member 127.0.0.3:31001 is up m31003| Thu Jan 17 17:03:42.343 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state SECONDARY m31001| Thu Jan 17 17:03:43.158 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:03:43.158 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:03:43.158 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:03:43.783 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:03:43.783 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:03:43.784 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:03:44.342 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31003| Thu Jan 17 17:03:44.342 [rsHealthPoll] Client::shutdown not called: rsHealthPoll { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:45Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 460, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:45Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 5, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:43Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 459, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:45Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:45Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:44Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:45Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 458, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:43Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:44Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:45Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:44Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:45Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 3, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 458, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31000| Thu Jan 17 17:03:50.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31002| Thu Jan 17 17:03:50.161 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31001| Thu Jan 17 17:03:50.162 [rsBackgroundSync] replSet syncing to: 127.0.0.2:31000 m31000| Thu Jan 17 17:03:50.162 [initandlisten] connection accepted from 127.0.0.4:46789 #104 (8 connections now open) m31000| Thu Jan 17 17:03:50.162 [initandlisten] connection accepted from 127.0.0.3:36537 #105 (9 connections now open) m31001| Thu Jan 17 17:03:50.163 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:03:50.163 [conn96] end connection 127.0.0.3:48594 (8 connections now open) m31000| Thu Jan 17 17:03:50.163 [initandlisten] connection accepted from 127.0.0.3:33331 #106 (9 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:50Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 465, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 10, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 464, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:50Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:50Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 463, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 9, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:49Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:50Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 8, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:48Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:49Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 463, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 17:03:50.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:03:54.164 [conn56] end connection 127.0.0.2:56557 (5 connections now open) m31002| Thu Jan 17 17:03:54.164 [initandlisten] connection accepted from 127.0.0.2:43103 #59 (6 connections now open) m31002| Thu Jan 17 17:03:55.162 [conn57] end connection 127.0.0.3:33181 (5 connections now open) m31002| Thu Jan 17 17:03:55.162 [initandlisten] connection accepted from 127.0.0.3:44824 #60 (6 connections now open) { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:55Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 470, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 15, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:53Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 469, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:55Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:55Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:54Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 468, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 14, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:53Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:54Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.4:31002" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:03:55Z"), "myState" : 2, "syncingTo" : "127.0.0.4:31002", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:54Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:55Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 13, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 468, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.4:31002", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.4:31002, it : 4, expected : 127.0.0.3:31001 m31001| Thu Jan 17 17:03:55.787 [conn56] end connection 127.0.0.4:40341 (3 connections now open) m31001| Thu Jan 17 17:03:55.788 [initandlisten] connection accepted from 127.0.0.4:38886 #61 (4 connections now open) m31003| Thu Jan 17 17:03:55.952 [rsBackgroundSync] replSet syncing to: 127.0.0.3:31001 m31002| Thu Jan 17 17:03:55.952 [conn50] end connection 127.0.0.5:36592 (5 connections now open) m31001| Thu Jan 17 17:03:55.952 [initandlisten] connection accepted from 127.0.0.5:40090 #62 (5 connections now open) m31003| Thu Jan 17 17:03:55.953 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.3:31001 m31002| Thu Jan 17 17:03:55.953 [conn51] end connection 127.0.0.5:49384 (4 connections now open) m31001| Thu Jan 17 17:03:55.953 [initandlisten] connection accepted from 127.0.0.5:47110 #63 (6 connections now open) m31000| Thu Jan 17 17:03:55.954 [conn83] end connection 127.0.0.3:41562 (8 connections now open) m31001| Thu Jan 17 17:03:55.954 [rsGhostSync] Socket recv() errno:104 Connection reset by peer 127.0.0.2:31000 m31001| Thu Jan 17 17:03:55.954 [rsGhostSync] SocketException: remote: 127.0.0.2:31000 error: 9001 socket exception [1] server [127.0.0.2:31000] m31001| Thu Jan 17 17:03:55.954 [rsGhostSync] Socket flush send() errno:32 Broken pipe 127.0.0.2:31000 m31001| Thu Jan 17 17:03:55.954 [rsGhostSync] caught exception (socket exception [SEND_ERROR] for 127.0.0.2:31000) in destructor (~PiggyBackData) m31001| Thu Jan 17 17:03:56.346 [conn57] end connection 127.0.0.5:47555 (5 connections now open) m31001| Thu Jan 17 17:03:56.346 [initandlisten] connection accepted from 127.0.0.5:48623 #64 (6 connections now open) m31000| Thu Jan 17 17:04:00.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:04:00Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 475, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "self" : true }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 20, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:04:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:04:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 474, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:04:00Z"), "myState" : 2, "syncingTo" : "127.0.0.2:31000", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:04:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 473, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "errmsg" : "syncing to: 127.0.0.2:31000", "self" : true }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 19, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:59Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.3:31001" } ], "ok" : 1 } { "set" : "testReplSet", "date" : ISODate("2013-01-17T22:04:00Z"), "myState" : 2, "syncingTo" : "127.0.0.3:31001", "members" : [ { "_id" : 0, "name" : "127.0.0.2:31000", "health" : -1, "state" : 6, "stateStr" : "UNKNOWN", "uptime" : 0, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "127.0.0.3:31001", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 2, "name" : "127.0.0.4:31002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18, "optime" : { "t" : 1358460155000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:02:35Z"), "lastHeartbeat" : ISODate("2013-01-17T22:03:58Z"), "lastHeartbeatRecv" : ISODate("2013-01-17T22:03:59Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: 127.0.0.2:31000" }, { "_id" : 3, "name" : "127.0.0.5:31003", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 473, "optime" : { "t" : 1358460220000, "i" : 1 }, "optimeDate" : ISODate("2013-01-17T22:03:40Z"), "errmsg" : "syncing to: 127.0.0.3:31001", "self" : true } ], "ok" : 1 } Syncing to: 127.0.0.3:31001, it : 4, expected : 127.0.0.3:31001 m31003| Thu Jan 17 17:04:00.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 17:04:09.165 [conn34] end connection 127.0.0.3:36180 (2 connections now open) m31003| Thu Jan 17 17:04:09.165 [initandlisten] connection accepted from 127.0.0.3:38669 #36 (3 connections now open) m31003| Thu Jan 17 17:04:09.790 [conn35] end connection 127.0.0.4:36786 (2 connections now open) m31003| Thu Jan 17 17:04:09.790 [initandlisten] connection accepted from 127.0.0.4:43259 #37 (3 connections now open) m31000| Thu Jan 17 17:04:10.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31001| Thu Jan 17 17:04:10.168 [conn60] end connection 127.0.0.2:48626 (5 connections now open) m31001| Thu Jan 17 17:04:10.168 [initandlisten] connection accepted from 127.0.0.2:55957 #65 (6 connections now open) m31003| Thu Jan 17 17:04:10.307 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31002| Thu Jan 17 17:04:10.349 [conn58] end connection 127.0.0.5:58462 (3 connections now open) m31002| Thu Jan 17 17:04:10.349 [initandlisten] connection accepted from 127.0.0.5:60565 #61 (4 connections now open) m31000| Thu Jan 17 17:04:11.165 [conn102] end connection 127.0.0.3:57462 (7 connections now open) m31000| Thu Jan 17 17:04:11.166 [initandlisten] connection accepted from 127.0.0.3:46813 #107 (8 connections now open) m31000| Thu Jan 17 17:04:11.790 [conn103] end connection 127.0.0.4:44306 (7 connections now open) m31000| Thu Jan 17 17:04:11.791 [initandlisten] connection accepted from 127.0.0.4:42207 #108 (8 connections now open) m31000| Thu Jan 17 17:04:20.147 [rsHealthPoll] replSet info 127.0.0.5:31003 is down (or slow to respond): m31000| Thu Jan 17 17:04:20.147 [rsHealthPoll] replSet member 127.0.0.5:31003 is now in state DOWN m31002| Thu Jan 17 17:04:20.163 [rsSyncNotifier] replset setting oplog notifier to 127.0.0.2:31000 m31000| Thu Jan 17 17:04:20.163 [conn92] end connection 127.0.0.4:38240 (7 connections now open) m31000| Thu Jan 17 17:04:20.163 [initandlisten] connection accepted from 127.0.0.4:39929 #109 (8 connections now open) m31003| Thu Jan 17 17:04:20.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:04:24.170 [conn59] end connection 127.0.0.2:43103 (3 connections now open) m31002| Thu Jan 17 17:04:24.170 [initandlisten] connection accepted from 127.0.0.2:57059 #62 (4 connections now open) m31002| Thu Jan 17 17:04:25.168 [conn60] end connection 127.0.0.3:44824 (3 connections now open) m31002| Thu Jan 17 17:04:25.169 [initandlisten] connection accepted from 127.0.0.3:53752 #63 (4 connections now open) m31001| Thu Jan 17 17:04:25.793 [conn61] end connection 127.0.0.4:38886 (5 connections now open) m31001| Thu Jan 17 17:04:25.793 [initandlisten] connection accepted from 127.0.0.4:57089 #66 (6 connections now open) m31001| Thu Jan 17 17:04:26.352 [conn64] end connection 127.0.0.5:48623 (5 connections now open) m31001| Thu Jan 17 17:04:26.352 [initandlisten] connection accepted from 127.0.0.5:58342 #67 (6 connections now open) m31000| Thu Jan 17 17:04:30.147 [rsHealthPoll] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:04:30.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31003| Thu Jan 17 17:04:30.353 [rsMgr] replSet I don't see a primary and I can't elect myself m31003| Thu Jan 17 17:04:39.171 [conn36] end connection 127.0.0.3:38669 (2 connections now open) m31003| Thu Jan 17 17:04:39.171 [initandlisten] connection accepted from 127.0.0.3:43550 #38 (3 connections now open) m31003| Thu Jan 17 17:04:39.796 [conn37] end connection 127.0.0.4:43259 (2 connections now open) m31003| Thu Jan 17 17:04:39.796 [initandlisten] connection accepted from 127.0.0.4:54917 #39 (3 connections now open) m31000| Thu Jan 17 17:04:40.147 [rsHealthPoll] Client::shutdown not called: rsHealthPoll m31001| Thu Jan 17 17:04:40.174 [conn65] end connection 127.0.0.2:55957 (5 connections now open) m31001| Thu Jan 17 17:04:40.174 [initandlisten] connection accepted from 127.0.0.2:44835 #68 (6 connections now open) m31003| Thu Jan 17 17:04:40.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31002| Thu Jan 17 17:04:40.355 [conn61] end connection 127.0.0.5:60565 (3 connections now open) m31002| Thu Jan 17 17:04:40.355 [initandlisten] connection accepted from 127.0.0.5:38743 #64 (4 connections now open) m31000| Thu Jan 17 17:04:41.171 [conn107] end connection 127.0.0.3:46813 (7 connections now open) m31000| Thu Jan 17 17:04:41.171 [initandlisten] connection accepted from 127.0.0.3:34106 #110 (8 connections now open) m31000| Thu Jan 17 17:04:41.796 [conn108] end connection 127.0.0.4:42207 (7 connections now open) m31000| Thu Jan 17 17:04:41.796 [initandlisten] connection accepted from 127.0.0.4:36858 #111 (8 connections now open) m31000| Thu Jan 17 17:04:50.147 [MultiCommandJob] couldn't connect to 127.0.0.5:31003: couldn't connect to server 127.0.0.5:31003 m31003| Thu Jan 17 17:04:50.307 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31003| Thu Jan 17 17:04:50.307 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31002| Thu Jan 17 17:04:54.176 [conn62] end connection 127.0.0.2:57059 (3 connections now open) m31002| Thu Jan 17 17:04:54.176 [initandlisten] connection accepted from 127.0.0.2:54880 #65 (4 connections now open) m31002| Thu Jan 17 17:04:55.174 [conn63] end connection 127.0.0.3:53752 (3 connections now open) m31002| Thu Jan 17 17:04:55.174 [initandlisten] connection accepted from 127.0.0.3:35110 #66 (4 connections now open) m31001| Thu Jan 17 17:04:55.799 [conn66] end connection 127.0.0.4:57089 (5 connections now open) m31001| Thu Jan 17 17:04:55.799 [initandlisten] connection accepted from 127.0.0.4:33237 #69 (6 connections now open) m31001| Thu Jan 17 17:04:56.358 [conn67] end connection 127.0.0.5:58342 (5 connections now open) m31001| Thu Jan 17 17:04:56.358 [initandlisten] connection accepted from 127.0.0.5:34326 #70 (6 connections now open) m31002| Thu Jan 17 17:04:59.800 [rsMgr] replSet I don't see a primary and I can't elect myself m31000| Thu Jan 17 17:05:00.147 [rsMgr] not electing self, 127.0.0.4:31002 would veto with '127.0.0.2:31000 is trying to elect itself but 127.0.0.2:31000 is already primary and more up-to-date' Thu Jan 17 17:05:02.278 javascript execution failed src/mongo/shell/utils.js:1261 [Finding master] timed out after 60000ms ( 31 tries ) pts['desc'] + ']' + " timed out after " + timeout + "ms ( " + tries + " tries ^ failed to load: /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js m31000| Thu Jan 17 17:05:02.278 got signal 15 (Terminated), will terminate after current cmd ends m31000| Thu Jan 17 17:05:02.278 [interruptThread] now exiting m31000| Thu Jan 17 17:05:02.278 dbexit: m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: going to close listening sockets... m31000| Thu Jan 17 17:05:02.278 [interruptThread] closing listening socket: 13 m31000| Thu Jan 17 17:05:02.278 [interruptThread] closing listening socket: 14 m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: going to flush diaglog... m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: going to close sockets... m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: waiting for fs preallocator... m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: lock for final commit... m31000| Thu Jan 17 17:05:02.278 [interruptThread] shutdown: final commit... m31002| Thu Jan 17 17:05:02.278 [conn65] end connection 127.0.0.2:54880 (3 connections now open) m31001| Thu Jan 17 17:05:02.279 [conn68] end connection 127.0.0.2:44835 (5 connections now open) m31002| Thu Jan 17 17:05:02.279 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31001| Thu Jan 17 17:05:02.279 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.2:31000 m31000| Thu Jan 17 17:05:02.280 [interruptThread] shutdown: closing all files... m31000| Thu Jan 17 17:05:02.280 [interruptThread] closeAllFiles() finished m31000| Thu Jan 17 17:05:02.280 [interruptThread] journalCleanup... m31000| Thu Jan 17 17:05:02.280 [interruptThread] removeJournalFiles m31000| Thu Jan 17 17:05:02.294 [interruptThread] shutdown: removing fs lock... m31000| Thu Jan 17 17:05:02.294 dbexit: really exiting now m31003| Thu Jan 17 17:05:02.307 [rsHealthPoll] couldn't connect to 127.0.0.2:31000: couldn't connect to server 127.0.0.2:31000 m31001| Thu Jan 17 17:05:03.176 [rsHealthPoll] DBClientCursor::init call() failed m31001| Thu Jan 17 17:05:03.176 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31001| Thu Jan 17 17:05:03.176 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31001| Thu Jan 17 17:05:03.176 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31001| Thu Jan 17 17:05:03.176 [rsMgr] replSet I don't see a primary and I can't elect myself m31001| Thu Jan 17 17:05:03.278 got signal 15 (Terminated), will terminate after current cmd ends m31001| Thu Jan 17 17:05:03.278 [interruptThread] now exiting m31001| Thu Jan 17 17:05:03.278 dbexit: m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: going to close listening sockets... m31001| Thu Jan 17 17:05:03.278 [interruptThread] closing listening socket: 16 m31001| Thu Jan 17 17:05:03.278 [interruptThread] closing listening socket: 17 m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: going to flush diaglog... m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: going to close sockets... m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: waiting for fs preallocator... m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: lock for final commit... m31001| Thu Jan 17 17:05:03.278 [interruptThread] shutdown: final commit... m31002| Thu Jan 17 17:05:03.279 [conn66] end connection 127.0.0.3:35110 (2 connections now open) m31003| Thu Jan 17 17:05:03.279 [conn38] end connection 127.0.0.3:43550 (2 connections now open) m31003| Thu Jan 17 17:05:03.279 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: 127.0.0.3:31001 m31001| Thu Jan 17 17:05:03.280 [interruptThread] shutdown: closing all files... m31001| Thu Jan 17 17:05:03.281 [interruptThread] closeAllFiles() finished m31001| Thu Jan 17 17:05:03.281 [interruptThread] journalCleanup... m31001| Thu Jan 17 17:05:03.281 [interruptThread] removeJournalFiles m31001| Thu Jan 17 17:05:03.302 [interruptThread] shutdown: removing fs lock... m31001| Thu Jan 17 17:05:03.302 dbexit: really exiting now m31002| Thu Jan 17 17:05:03.800 [rsHealthPoll] DBClientCursor::init call() failed m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] DBClientCursor::init call() failed m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replset info 127.0.0.2:31000 heartbeat failed, retrying m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replset info 127.0.0.3:31001 heartbeat failed, retrying m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replSet info 127.0.0.2:31000 is down (or slow to respond): m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replSet info 127.0.0.3:31001 is down (or slow to respond): m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replSet member 127.0.0.2:31000 is now in state DOWN m31002| Thu Jan 17 17:05:03.801 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state DOWN m31002| Thu Jan 17 17:05:04.278 got signal 15 (Terminated), will terminate after current cmd ends m31002| Thu Jan 17 17:05:04.278 [interruptThread] now exiting m31002| Thu Jan 17 17:05:04.278 dbexit: m31002| Thu Jan 17 17:05:04.278 [interruptThread] shutdown: going to close listening sockets... m31002| Thu Jan 17 17:05:04.278 [interruptThread] closing listening socket: 19 m31002| Thu Jan 17 17:05:04.278 [interruptThread] closing listening socket: 20 m31002| Thu Jan 17 17:05:04.278 [interruptThread] shutdown: going to flush diaglog... m31002| Thu Jan 17 17:05:04.278 [interruptThread] shutdown: going to close sockets... m31002| Thu Jan 17 17:05:04.278 [interruptThread] shutdown: waiting for fs preallocator... m31002| Thu Jan 17 17:05:04.278 [interruptThread] shutdown: lock for final commit... m31002| Thu Jan 17 17:05:04.279 [interruptThread] shutdown: final commit... m31003| Thu Jan 17 17:05:04.279 [conn39] end connection 127.0.0.4:54917 (1 connection now open) m31002| Thu Jan 17 17:05:04.280 [interruptThread] shutdown: closing all files... m31002| Thu Jan 17 17:05:04.281 [interruptThread] closeAllFiles() finished m31002| Thu Jan 17 17:05:04.281 [interruptThread] journalCleanup... m31002| Thu Jan 17 17:05:04.281 [interruptThread] removeJournalFiles m31002| Thu Jan 17 17:05:04.293 [interruptThread] shutdown: removing fs lock... m31002| Thu Jan 17 17:05:04.293 dbexit: really exiting now m31003| Thu Jan 17 17:05:04.359 [rsHealthPoll] DBClientCursor::init call() failed m31003| Thu Jan 17 17:05:04.359 [rsHealthPoll] DBClientCursor::init call() failed m31003| Thu Jan 17 17:05:04.359 [rsHealthPoll] replset info 127.0.0.3:31001 heartbeat failed, retrying m31003| Thu Jan 17 17:05:04.359 [rsHealthPoll] replset info 127.0.0.4:31002 heartbeat failed, retrying m31003| Thu Jan 17 17:05:04.360 [rsHealthPoll] replSet info 127.0.0.3:31001 is down (or slow to respond): m31003| Thu Jan 17 17:05:04.360 [rsHealthPoll] replSet info 127.0.0.4:31002 is down (or slow to respond): m31003| Thu Jan 17 17:05:04.360 [rsHealthPoll] replSet member 127.0.0.3:31001 is now in state DOWN m31003| Thu Jan 17 17:05:04.360 [rsHealthPoll] replSet member 127.0.0.4:31002 is now in state DOWN m31003| Thu Jan 17 17:05:05.278 got signal 15 (Terminated), will terminate after current cmd ends m31003| Thu Jan 17 17:05:05.279 [interruptThread] now exiting m31003| Thu Jan 17 17:05:05.279 dbexit: m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: going to close listening sockets... m31003| Thu Jan 17 17:05:05.279 [interruptThread] closing listening socket: 22 m31003| Thu Jan 17 17:05:05.279 [interruptThread] closing listening socket: 23 m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: going to flush diaglog... m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: going to close sockets... m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: waiting for fs preallocator... m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: lock for final commit... m31003| Thu Jan 17 17:05:05.279 [interruptThread] shutdown: final commit... m31003| Thu Jan 17 17:05:05.280 [interruptThread] shutdown: closing all files... m31003| Thu Jan 17 17:05:05.281 [interruptThread] closeAllFiles() finished m31003| Thu Jan 17 17Thu Jan 17 17:05:06.281 [conn2] end connection 127.0.0.1:56757 (0 connections now open) 9.0103 minutes Thu Jan 17 17:05:06.282 got signal 15 (Terminated), will terminate after current cmd ends Thu Jan 17 17:05:06.282 [interruptThread] now exiting Thu Jan 17 17:05:06.282 dbexit: Thu Jan 17 17:05:06.282 [interruptThread] shutdown: going to close listening sockets... Thu Jan 17 17:05:06.282 [interruptThread] closing listening socket: 9 Thu Jan 17 17:05:06.282 [interruptThread] closing listening socket: 10 Thu Jan 17 17:05:06.282 [interruptThread] closing listening socket: 11 Thu Jan 17 17:05:06.282 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Thu Jan 17 17:05:06.282 [interruptThread] shutdown: going to flush diaglog... Thu Jan 17 17:05:06.283 [interruptThread] shutdown: going to close sockets... Thu Jan 17 17:05:06.283 [interruptThread] shutdown: waiting for fs preallocator... Thu Jan 17 17:05:06.283 [interruptThread] shutdown: lock for final commit... Thu Jan 17 17:05:06.283 [interruptThread] shutdown: final commit... Thu Jan 17 17:05:06.283 [interruptThread] shutdown: closing all files... Thu Jan 17 17:05:06.284 [interruptThread] closeAllFiles() finished Thu Jan 17 17:05:06.284 [interruptThread] journalCleanup... Thu Jan 17 17:05:06.284 [interruptThread] removeJournalFiles Thu Jan 17 17:05:06.302 [interruptThread] shutdown: removing fs lock... Thu Jan 17 17:05:06.302 dbexit: really exiting now test /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js exited with status 253 0 tests succeeded The following tests failed (with exit code): /home/gregorv/Workspaces/10Gen/mongo/jstests/replsets/sync_change_source.js 253