Mon Oct 15 21:24:55 [conn49] end connection 127.0.0.1:35257 (0 connections now open) MongoDB shell version: 2.3.0-pre- null stepdown2.js ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testSet", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "testSet" } } ReplSetTest Starting.... Resetting db path '/data/db/testSet-0' Mon Oct 15 21:24:56 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31000 --noprealloc --smallfiles --rest --replSet testSet --dbpath /data/db/testSet-0 m31000| note: noprealloc may hurt performance in many applications m31000| Mon Oct 15 21:24:56 [initandlisten] MongoDB starting : pid=14690 port=31000 dbpath=/data/db/testSet-0 64-bit host=domU-12-31-39-16-30-A2 m31000| Mon Oct 15 21:24:56 [initandlisten] m31000| Mon Oct 15 21:24:56 [initandlisten] ** NOTE: This is a development version (2.3.0-pre-) of MongoDB. m31000| Mon Oct 15 21:24:56 [initandlisten] ** Not recommended for production. m31000| Mon Oct 15 21:24:56 [initandlisten] m31000| Mon Oct 15 21:24:56 [initandlisten] db version v2.3.0-pre-, pdfile version 4.5 m31000| Mon Oct 15 21:24:56 [initandlisten] git version: 2e287063e6ec4a08b716288cc0ecedbfad4f3d3c m31000| Mon Oct 15 21:24:56 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31000| Mon Oct 15 21:24:56 [initandlisten] options: { dbpath: "/data/db/testSet-0", noprealloc: true, oplogSize: 40, port: 31000, replSet: "testSet", rest: true, smallfiles: true } m31000| Mon Oct 15 21:24:56 [initandlisten] journal dir=/data/db/testSet-0/journal m31000| Mon Oct 15 21:24:56 [initandlisten] recover : no journal files present, no recovery needed m31000| Mon Oct 15 21:24:56 [initandlisten] waiting for connections on port 31000 m31000| Mon Oct 15 21:24:56 [websvr] admin web console waiting for connections on port 32000 m31000| Mon Oct 15 21:24:56 [initandlisten] connection accepted from 127.0.0.1:58082 #1 (1 connection now open) m31000| Mon Oct 15 21:24:56 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Mon Oct 15 21:24:56 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "testSet", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "testSet" } } ReplSetTest Starting.... Resetting db path '/data/db/testSet-1' Mon Oct 15 21:24:56 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31001 --noprealloc --smallfiles --rest --replSet testSet --dbpath /data/db/testSet-1 m31000| Mon Oct 15 21:24:56 [initandlisten] connection accepted from 127.0.0.1:58083 #2 (2 connections now open) m31001| note: noprealloc may hurt performance in many applications m31001| Mon Oct 15 21:24:56 [initandlisten] MongoDB starting : pid=14738 port=31001 dbpath=/data/db/testSet-1 64-bit host=domU-12-31-39-16-30-A2 m31001| Mon Oct 15 21:24:56 [initandlisten] m31001| Mon Oct 15 21:24:56 [initandlisten] ** NOTE: This is a development version (2.3.0-pre-) of MongoDB. m31001| Mon Oct 15 21:24:56 [initandlisten] ** Not recommended for production. m31001| Mon Oct 15 21:24:56 [initandlisten] m31001| Mon Oct 15 21:24:56 [initandlisten] db version v2.3.0-pre-, pdfile version 4.5 m31001| Mon Oct 15 21:24:56 [initandlisten] git version: 2e287063e6ec4a08b716288cc0ecedbfad4f3d3c m31001| Mon Oct 15 21:24:56 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31001| Mon Oct 15 21:24:56 [initandlisten] options: { dbpath: "/data/db/testSet-1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "testSet", rest: true, smallfiles: true } m31001| Mon Oct 15 21:24:56 [initandlisten] journal dir=/data/db/testSet-1/journal m31001| Mon Oct 15 21:24:56 [initandlisten] recover : no journal files present, no recovery needed m31001| Mon Oct 15 21:24:56 [websvr] admin web console waiting for connections on port 32001 m31001| Mon Oct 15 21:24:56 [initandlisten] waiting for connections on port 31001 m31001| Mon Oct 15 21:24:56 [initandlisten] connection accepted from 127.0.0.1:40381 #1 (1 connection now open) m31001| Mon Oct 15 21:24:56 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Mon Oct 15 21:24:56 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31000, connection to domU-12-31-39-16-30-A2:31001 ] { "replSetInitiate" : { "_id" : "testSet", "members" : [ { "_id" : 0, "host" : "domU-12-31-39-16-30-A2:31000" }, { "_id" : 1, "host" : "domU-12-31-39-16-30-A2:31001" } ] } } m31000| Mon Oct 15 21:24:56 [conn2] replSet replSetInitiate admin command received from client m31000| Mon Oct 15 21:24:56 [conn2] replSet replSetInitiate config object parses ok, 2 members specified m31001| Mon Oct 15 21:24:56 [initandlisten] connection accepted from 127.0.0.1:40382 #2 (2 connections now open) m31001| Mon Oct 15 21:24:56 [initandlisten] connection accepted from 127.0.0.1:40383 #3 (3 connections now open) m31000| Mon Oct 15 21:24:56 [conn2] replSet replSetInitiate all members seem up m31000| Mon Oct 15 21:24:56 [conn2] ****** m31000| Mon Oct 15 21:24:56 [conn2] creating replication oplog of size: 40MB... m31000| Mon Oct 15 21:24:56 [FileAllocator] allocating new datafile /data/db/testSet-0/local.ns, filling with zeroes... m31000| Mon Oct 15 21:24:56 [FileAllocator] creating directory /data/db/testSet-0/_tmp m31000| Mon Oct 15 21:24:56 [FileAllocator] done allocating datafile /data/db/testSet-0/local.ns, size: 16MB, took 0.217 secs m31000| Mon Oct 15 21:24:56 [FileAllocator] allocating new datafile /data/db/testSet-0/local.0, filling with zeroes... m31000| Mon Oct 15 21:24:58 [FileAllocator] done allocating datafile /data/db/testSet-0/local.0, size: 64MB, took 1.142 secs m31000| Mon Oct 15 21:24:58 [conn2] ****** m31000| Mon Oct 15 21:24:58 [conn2] replSet info saving a newer config version to local.system.replset m31000| Mon Oct 15 21:24:58 [conn2] replSet saveConfigLocally done m31000| Mon Oct 15 21:24:58 [conn2] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Mon Oct 15 21:24:58 [conn2] command admin.$cmd command: { replSetInitiate: { _id: "testSet", members: [ { _id: 0.0, host: "domU-12-31-39-16-30-A2:31000" }, { _id: 1.0, host: "domU-12-31-39-16-30-A2:31001" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1416864 reslen:112 1417ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Mon Oct 15 21:25:06 [rsStart] replSet I am domU-12-31-39-16-30-A2:31000 m31000| Mon Oct 15 21:25:06 [rsStart] replSet STARTUP2 m31000| Mon Oct 15 21:25:06 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31000| Mon Oct 15 21:25:06 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is up m31001| Mon Oct 15 21:25:06 [rsStart] trying to contact domU-12-31-39-16-30-A2:31000 m31000| Mon Oct 15 21:25:06 [initandlisten] connection accepted from 127.0.0.1:58099 #3 (3 connections now open) m31001| Mon Oct 15 21:25:06 [rsStart] replSet I am domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:25:06 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Mon Oct 15 21:25:06 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Mon Oct 15 21:25:06 [FileAllocator] allocating new datafile /data/db/testSet-1/local.ns, filling with zeroes... m31001| Mon Oct 15 21:25:06 [FileAllocator] creating directory /data/db/testSet-1/_tmp m31001| Mon Oct 15 21:25:06 [FileAllocator] done allocating datafile /data/db/testSet-1/local.ns, size: 16MB, took 0.261 secs m31001| Mon Oct 15 21:25:06 [FileAllocator] allocating new datafile /data/db/testSet-1/local.0, filling with zeroes... m31001| Mon Oct 15 21:25:07 [FileAllocator] done allocating datafile /data/db/testSet-1/local.0, size: 16MB, took 0.231 secs m31001| Mon Oct 15 21:25:07 [rsStart] replSet saveConfigLocally done m31001| Mon Oct 15 21:25:07 [rsStart] replSet STARTUP2 m31001| Mon Oct 15 21:25:07 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31001| Mon Oct 15 21:25:07 [rsSync] ****** m31001| Mon Oct 15 21:25:07 [rsSync] creating replication oplog of size: 40MB... m31001| Mon Oct 15 21:25:07 [FileAllocator] allocating new datafile /data/db/testSet-1/local.1, filling with zeroes... m31000| Mon Oct 15 21:25:07 [rsSync] replSet SECONDARY m31001| Mon Oct 15 21:25:08 [FileAllocator] done allocating datafile /data/db/testSet-1/local.1, size: 64MB, took 0.929 secs m31001| Mon Oct 15 21:25:08 [rsSync] ****** m31001| Mon Oct 15 21:25:08 [rsSync] replSet initial sync pending m31001| Mon Oct 15 21:25:08 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Mon Oct 15 21:25:08 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31001 thinks that we are down m31000| Mon Oct 15 21:25:08 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state STARTUP2 m31000| Mon Oct 15 21:25:08 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31001 would veto m31001| Mon Oct 15 21:25:08 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is up m31001| Mon Oct 15 21:25:08 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is now in state SECONDARY m31000| Mon Oct 15 21:25:14 [rsMgr] replSet info electSelf 0 m31001| Mon Oct 15 21:25:14 [conn3] replSet RECOVERING m31001| Mon Oct 15 21:25:14 [conn3] replSet info voting yea for domU-12-31-39-16-30-A2:31000 (0) m31000| Mon Oct 15 21:25:14 [rsMgr] replSet PRIMARY m31001| Mon Oct 15 21:25:14 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is now in state PRIMARY do a write m31000| Mon Oct 15 21:25:16 [FileAllocator] allocating new datafile /data/db/testSet-0/foo.ns, filling with zeroes... m31000| Mon Oct 15 21:25:16 [FileAllocator] done allocating datafile /data/db/testSet-0/foo.ns, size: 16MB, took 0.255 secs m31000| Mon Oct 15 21:25:16 [FileAllocator] allocating new datafile /data/db/testSet-0/foo.0, filling with zeroes... m31000| Mon Oct 15 21:25:16 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state RECOVERING m31000| Mon Oct 15 21:25:16 [FileAllocator] done allocating datafile /data/db/testSet-0/foo.0, size: 16MB, took 0.202 secs m31000| Mon Oct 15 21:25:16 [conn2] build index foo.bar { _id: 1 } m31000| Mon Oct 15 21:25:16 [conn2] build index done. scanned 0 total records. 0.024 secs m31000| Mon Oct 15 21:25:16 [conn2] insert foo.bar keyUpdates:0 locks(micros) w:493131 492ms ReplSetTest Timestamp(1350350716000, 1) ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31001 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31001 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31001 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31001 to have an oplog built. m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync pending m31001| Mon Oct 15 21:25:24 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31000 m31000| Mon Oct 15 21:25:24 [initandlisten] connection accepted from 127.0.0.1:58126 #4 (4 connections now open) m31001| Mon Oct 15 21:25:24 [rsSync] build index local.me { _id: 1 } m31001| Mon Oct 15 21:25:24 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync drop all databases m31001| Mon Oct 15 21:25:24 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync clone all databases m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync cloning db: foo m31000| Mon Oct 15 21:25:24 [initandlisten] connection accepted from 127.0.0.1:58127 #5 (5 connections now open) m31001| Mon Oct 15 21:25:24 [FileAllocator] allocating new datafile /data/db/testSet-1/foo.ns, filling with zeroes... m31001| Mon Oct 15 21:25:24 [FileAllocator] done allocating datafile /data/db/testSet-1/foo.ns, size: 16MB, took 0.31 secs m31001| Mon Oct 15 21:25:24 [FileAllocator] allocating new datafile /data/db/testSet-1/foo.0, filling with zeroes... ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31001 to have an oplog built. m31001| Mon Oct 15 21:25:24 [FileAllocator] done allocating datafile /data/db/testSet-1/foo.0, size: 16MB, took 0.277 secs m31001| Mon Oct 15 21:25:24 [rsSync] build index foo.bar { _id: 1 } m31000| Mon Oct 15 21:25:24 [conn5] end connection 127.0.0.1:58127 (4 connections now open) m31001| Mon Oct 15 21:25:24 [rsSync] fastBuildIndex dupsToDrop:0 m31001| Mon Oct 15 21:25:24 [rsSync] build index done. scanned 1 total records. 0.021 secs m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync data copy, starting syncup m31001| Mon Oct 15 21:25:24 [rsSync] oplog sync 1 of 3 m31001| Mon Oct 15 21:25:24 [rsSync] oplog sync 2 of 3 m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync building indexes m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync cloning indexes for : foo m31000| Mon Oct 15 21:25:24 [initandlisten] connection accepted from 127.0.0.1:58131 #6 (5 connections now open) m31000| Mon Oct 15 21:25:24 [conn6] end connection 127.0.0.1:58131 (4 connections now open) m31001| Mon Oct 15 21:25:24 [rsSync] oplog sync 3 of 3 m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync finishing up m31000| Mon Oct 15 21:25:24 [conn4] end connection 127.0.0.1:58126 (3 connections now open) m31001| Mon Oct 15 21:25:24 [rsSync] replSet set minValid=507cb77c:1 m31001| Mon Oct 15 21:25:24 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Mon Oct 15 21:25:24 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Mon Oct 15 21:25:24 [rsSync] replSet initial sync done m31001| Mon Oct 15 21:25:25 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31000 m31000| Mon Oct 15 21:25:25 [initandlisten] connection accepted from 127.0.0.1:58132 #7 (4 connections now open) m31001| Mon Oct 15 21:25:25 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31000 m31000| Mon Oct 15 21:25:25 [initandlisten] connection accepted from 127.0.0.1:58133 #8 (5 connections now open) { "ts" : Timestamp(1350350716000, 1), "h" : NumberLong("6361386408405368849"), "v" : 2, "op" : "i", "ns" : "foo.bar", "o" : { "_id" : ObjectId("507cb77cceff06584d2ac626"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31001 is 1350350716000:1 and latest is 1350350716000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31001 is 1 ReplSetTest await synced=true lock secondary m31001| Mon Oct 15 21:25:26 [conn2] CMD fsync: sync:1 lock:1 m31001| Mon Oct 15 21:25:26 [fsyncLockWorker] removeJournalFiles m31001| Mon Oct 15 21:25:26 [conn2] db is now locked for snapshotting, no writes allowed. db.fsyncUnlock() to unlock m31001| Mon Oct 15 21:25:26 [conn2] For more info see http://dochub.mongodb.org/core/fsynccommand { "info" : "now locked against writes, use db.fsyncUnlock() to unlock", "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand", "ok" : 1 } waiting 11ish seconds m31000| Mon Oct 15 21:25:26 [slaveTracking] build index local.slaves { _id: 1 } m31000| Mon Oct 15 21:25:26 [slaveTracking] build index done. scanned 0 total records. 0 secs m31001| Mon Oct 15 21:25:26 [rsSync] replSet SECONDARY m31000| Mon Oct 15 21:25:28 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state SECONDARY m31001| Mon Oct 15 21:25:34 [conn3] end connection 127.0.0.1:40383 (2 connections now open) m31001| Mon Oct 15 21:25:34 [initandlisten] connection accepted from 127.0.0.1:40438 #4 (3 connections now open) m31000| Mon Oct 15 21:25:36 [conn3] end connection 127.0.0.1:58099 (4 connections now open) m31000| Mon Oct 15 21:25:36 [initandlisten] connection accepted from 127.0.0.1:58148 #9 (5 connections now open) do stepdown that should not work { "closest" : NumberLong(1350350716), "difference" : NumberLong(23), "errmsg" : "no secondaries within 10 seconds of my optime", "ok" : 0 } do stepdown that should work Mon Oct 15 21:25:40 DBClientCursor::init call() failed Mon Oct 15 21:25:40 query failed : admin.$cmd { replSetStepDown: 50.0, force: true } to: 127.0.0.1:31000 Error: error doing query: failed Mon Oct 15 21:25:40 trying reconnect to 127.0.0.1:31000 Mon Oct 15 21:25:40 reconnect 127.0.0.1:31000 ok unlock { "ok" : 1, "info" : "unlock completed" } reset stepped down time m31000| Mon Oct 15 21:25:40 [conn2] replSet info stepping down as primary secs=50 m31000| Mon Oct 15 21:25:40 [conn2] replSet relinquishing primary state m31000| Mon Oct 15 21:25:40 [conn2] replSet SECONDARY m31000| Mon Oct 15 21:25:40 [conn2] replSet closing client sockets after relinquishing primary m31000| Mon Oct 15 21:25:40 [conn2] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:58083] m31000| Mon Oct 15 21:25:40 [initandlisten] connection accepted from 127.0.0.1:58149 #10 (5 connections now open) m31000| Mon Oct 15 21:25:40 [conn10] replSet info 'unfreezing' m31001| Mon Oct 15 21:25:40 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31000 m31001| Mon Oct 15 21:25:40 [conn2] command: unlock requested m31000| Mon Oct 15 21:25:40 [conn8] end connection 127.0.0.1:58133 (4 connections now open) m31001| Mon Oct 15 21:25:40 [rsSyncNotifier] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31000 m31001| Mon Oct 15 21:25:40 [rsSyncNotifier] SocketException: remote: 127.0.0.1:31000 error: 9001 socket exception [1] server [127.0.0.1:31000] m31001| Mon Oct 15 21:25:40 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31000 m31001| Mon Oct 15 21:25:41 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31000 m31001| Mon Oct 15 21:25:41 [rsSyncNotifier] Socket flush send() errno:32 Broken pipe 127.0.0.1:31000 m31001| Mon Oct 15 21:25:41 [rsSyncNotifier] caught exception (socket exception [SEND_ERROR] for 127.0.0.1:31000) in destructor (~PiggyBackData) m31000| Mon Oct 15 21:25:41 [initandlisten] connection accepted from 127.0.0.1:58153 #11 (5 connections now open) m31001| Mon Oct 15 21:25:42 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is now in state SECONDARY m31001| Mon Oct 15 21:25:42 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31000 would veto m31000| Mon Oct 15 21:25:44 [conn7] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:58132] m31000| Mon Oct 15 21:25:46 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31001 would veto m31001| Mon Oct 15 21:25:49 [rsMgr] replSet info electSelf 1 m31000| Mon Oct 15 21:25:49 [conn9] replSet info voting yea for domU-12-31-39-16-30-A2:31001 (1) m31001| Mon Oct 15 21:25:49 [rsMgr] replSet PRIMARY m31000| Mon Oct 15 21:25:50 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state PRIMARY m31000| Mon Oct 15 21:25:50 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:25:50 [initandlisten] connection accepted from 127.0.0.1:40457 #5 (4 connections now open) make 1 config with priorities make 2 m31001| Mon Oct 15 21:25:50 [conn2] replSet replSetReconfig config object parses ok, 2 members specified Mon Oct 15 21:25:50 DBClientCursor::init call() failed Mon Oct 15 21:25:50 query failed : admin.$cmd { replSetReconfig: { _id: "testSet", version: 2, members: [ { _id: 0, host: "domU-12-31-39-16-30-A2:31000", priority: 2.0, votes: 0.0 }, { _id: 1, host: "domU-12-31-39-16-30-A2:31001", priority: 1.0 } ] } } to: 127.0.0.1:31001 Error: error doing query: failed await Mon Oct 15 21:25:50 trying reconnect to 127.0.0.1:31001 Mon Oct 15 21:25:50 reconnect 127.0.0.1:31001 ok m31001| Mon Oct 15 21:25:50 [conn2] replSet replSetReconfig [2] m31001| Mon Oct 15 21:25:50 [conn2] replSet info saving a newer config version to local.system.replset m31001| Mon Oct 15 21:25:50 [conn2] replSet saveConfigLocally done m31001| Mon Oct 15 21:25:50 [conn2] replSet relinquishing primary state m31001| Mon Oct 15 21:25:50 [conn2] replSet SECONDARY m31001| Mon Oct 15 21:25:50 [conn2] replSet closing client sockets after relinquishing primary m31001| Mon Oct 15 21:25:50 [conn2] replSet PRIMARY m31001| Mon Oct 15 21:25:50 [conn2] replSet replSetReconfig new config saved locally m31001| Mon Oct 15 21:25:50 [conn2] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40382] m31001| Mon Oct 15 21:25:50 [conn5] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40457] m31001| Mon Oct 15 21:25:50 [initandlisten] connection accepted from 127.0.0.1:40459 #6 (3 connections now open) m31001| Mon Oct 15 21:25:50 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is up m31001| Mon Oct 15 21:25:50 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is now in state SECONDARY m31001| Mon Oct 15 21:25:50 [rsMgr] stepping down domU-12-31-39-16-30-A2:31001 (priority 1), domU-12-31-39-16-30-A2:31000 is priority 2 and 0 seconds behind m31001| Mon Oct 15 21:25:50 [rsMgr] replSet relinquishing primary state m31000| Mon Oct 15 21:25:50 [conn11] end connection 127.0.0.1:58153 (3 connections now open) m31000| Mon Oct 15 21:25:50 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:25:50 [rsMgr] replSet SECONDARY m31001| Mon Oct 15 21:25:50 [rsMgr] replSet closing client sockets after relinquishing primary Mon Oct 15 21:25:50 DBClientCursor::init call() failed Mon Oct 15 21:25:50 query failed : admin.$cmd { ismaster: 1.0 } to: 127.0.0.1:31001 ReplSetTest Could not call ismaster on node 1 m31001| Mon Oct 15 21:25:50 [conn6] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40459] m31000| Mon Oct 15 21:25:51 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:25:51 [initandlisten] connection accepted from 127.0.0.1:40462 #7 (3 connections now open) m31000| Mon Oct 15 21:25:51 [rsSyncNotifier] build index local.me { _id: 1 } m31000| Mon Oct 15 21:25:51 [rsSyncNotifier] build index done. scanned 0 total records. 0 secs m31000| Mon Oct 15 21:25:52 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state SECONDARY m31000| Mon Oct 15 21:25:52 [rsMgr] replset msgReceivedNewConfig version: version: 2 m31000| Mon Oct 15 21:25:52 [rsMgr] replSet info saving a newer config version to local.system.replset m31000| Mon Oct 15 21:25:52 [rsMgr] replSet saveConfigLocally done m31000| Mon Oct 15 21:25:52 [rsMgr] replSet replSetReconfig new config saved locally m31000| Mon Oct 15 21:25:52 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is up m31000| Mon Oct 15 21:25:52 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state SECONDARY m31001| Mon Oct 15 21:25:52 [rsHealthPoll] ERROR: Client::shutdown not called: rsHealthPoll Mon Oct 15 21:25:52 trying reconnect to 127.0.0.1:31001 Mon Oct 15 21:25:52 reconnect 127.0.0.1:31001 ok m31001| Mon Oct 15 21:25:52 [initandlisten] connection accepted from 127.0.0.1:40464 #8 (4 connections now open) m31000| Mon Oct 15 21:25:54 [rsHealthPoll] ERROR: Client::shutdown not called: rsHealthPoll m31001| Mon Oct 15 21:25:56 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31000 would veto m31000| Mon Oct 15 21:26:00 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31001 m31000| Mon Oct 15 21:26:00 [rsSync] build index local.replset.minvalid { _id: 1 } m31000| Mon Oct 15 21:26:00 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Mon Oct 15 21:26:00 [initandlisten] connection accepted from 127.0.0.1:40474 #9 (5 connections now open) m31000| Mon Oct 15 21:26:00 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:26:00 [conn7] end connection 127.0.0.1:40462 (4 connections now open) m31001| Mon Oct 15 21:26:00 [initandlisten] connection accepted from 127.0.0.1:40475 #10 (5 connections now open) m31001| Mon Oct 15 21:26:01 [slaveTracking] build index local.slaves { _id: 1 } m31001| Mon Oct 15 21:26:01 [slaveTracking] build index done. scanned 0 total records. 0 secs m31000| Mon Oct 15 21:26:03 [conn9] replSet info voting yea for domU-12-31-39-16-30-A2:31001 (1) m31001| Mon Oct 15 21:26:03 [rsMgr] replSet info electSelf 1 m31001| Mon Oct 15 21:26:03 [rsMgr] replSet PRIMARY m31000| Mon Oct 15 21:26:04 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state PRIMARY m31000| Mon Oct 15 21:26:04 [conn9] end connection 127.0.0.1:58148 (2 connections now open) m31000| Mon Oct 15 21:26:04 [initandlisten] connection accepted from 127.0.0.1:58188 #12 (3 connections now open) ReplSetTest Timestamp(1350350750000, 1) { "ts" : Timestamp(1350350750000, 1), "h" : NumberLong("-1215809815505824934"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 2 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31000 is 1350350750000:1 and latest is 1350350750000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31000 is 14 ReplSetTest await synced=true master is now connection to domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:26:04 [conn8] replSet info stepping down as primary secs=100 m31001| Mon Oct 15 21:26:04 [conn8] replSet relinquishing primary state m31001| Mon Oct 15 21:26:04 [conn8] replSet SECONDARY m31001| Mon Oct 15 21:26:04 [conn8] replSet closing client sockets after relinquishing primary m31001| Mon Oct 15 21:26:04 [conn8] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40464] Mon Oct 15 21:26:04 DBClientCursor::init call() failed Mon Oct 15 21:26:04 query failed : admin.$cmd { replSetStepDown: 100.0, force: true } to: 127.0.0.1:31001 Error: error doing query: failed get new master (31001) Mon Oct 15 21:26:04 trying reconnect to 127.0.0.1:31001 Mon Oct 15 21:26:04 reconnect 127.0.0.1:31001 ok m31001| Mon Oct 15 21:26:04 [initandlisten] connection accepted from 127.0.0.1:40482 #11 (5 connections now open) m31000| Mon Oct 15 21:26:04 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31001 m31000| Mon Oct 15 21:26:04 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31001 m31001| Mon Oct 15 21:26:06 [conn9] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40474] m31001| Mon Oct 15 21:26:06 [conn10] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40475] m31001| Mon Oct 15 21:26:06 [conn4] end connection 127.0.0.1:40438 (2 connections now open) m31001| Mon Oct 15 21:26:06 [initandlisten] connection accepted from 127.0.0.1:40486 #12 (3 connections now open) m31000| Mon Oct 15 21:26:06 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state SECONDARY m31000| Mon Oct 15 21:26:06 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31000| Mon Oct 15 21:26:12 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31000| Mon Oct 15 21:26:18 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31000| Mon Oct 15 21:26:24 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31000| Mon Oct 15 21:26:30 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31000| Mon Oct 15 21:26:34 [conn12] end connection 127.0.0.1:58188 (2 connections now open) m31000| Mon Oct 15 21:26:34 [initandlisten] connection accepted from 127.0.0.1:46125 #13 (3 connections now open) m31001| Mon Oct 15 21:26:36 [conn12] end connection 127.0.0.1:40486 (2 connections now open) m31001| Mon Oct 15 21:26:36 [initandlisten] connection accepted from 127.0.0.1:48858 #13 (3 connections now open) m31000| Mon Oct 15 21:26:36 [rsMgr] replSet info electSelf 0 m31000| Mon Oct 15 21:26:36 [rsMgr] replSet PRIMARY m31001| Mon Oct 15 21:26:36 [conn13] replSet info voting yea for domU-12-31-39-16-30-A2:31000 (0) m31001| Mon Oct 15 21:26:36 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31000 is now in state PRIMARY check shutdown command on 31000 m31001| Mon Oct 15 21:26:36 [conn11] terminating, shutdown command received m31001| Mon Oct 15 21:26:36 dbexit: shutdown called m31001| Mon Oct 15 21:26:36 [conn11] shutdown: going to close listening sockets... m31001| Mon Oct 15 21:26:36 [conn11] closing listening socket: 14 m31001| Mon Oct 15 21:26:36 [conn11] closing listening socket: 15 m31001| Mon Oct 15 21:26:36 [conn11] closing listening socket: 16 m31001| Mon Oct 15 21:26:36 [conn11] removing socket file: /tmp/mongodb-31001.sock m31001| Mon Oct 15 21:26:36 [conn11] shutdown: going to flush diaglog... m31001| Mon Oct 15 21:26:36 [conn11] shutdown: going to close sockets... m31001| Mon Oct 15 21:26:36 [conn11] shutdown: waiting for fs preallocator... m31001| Mon Oct 15 21:26:36 [conn11] shutdown: lock for final commit... m31001| Mon Oct 15 21:26:36 [conn11] shutdown: final commit... m31001| Mon Oct 15 21:26:36 [conn11] shutdown: closing all files... m31001| Mon Oct 15 21:26:36 [conn11] closeAllFiles() finished m31001| Mon Oct 15 21:26:36 [conn11] journalCleanup... m31001| Mon Oct 15 21:26:36 [conn11] removeJournalFiles m31001| Mon Oct 15 21:26:36 [conn1] end connection 127.0.0.1:40381 (2 connections now open) m31001| Mon Oct 15 21:26:36 [conn11] shutdown: removing fs lock... m31001| Mon Oct 15 21:26:36 dbexit: really exiting now Mon Oct 15 21:26:36 DBClientCursor::init call() failed Mon Oct 15 21:26:36 query failed : admin.$cmd { shutdown: 1.0 } to: 127.0.0.1:31001 Error: error doing query: failed wait until 31001 knows 31000 is down m31000| Mon Oct 15 21:26:36 [conn13] end connection 127.0.0.1:46125 (2 connections now open) m31000| Mon Oct 15 21:26:38 [rsHealthPoll] DBClientCursor::init call() failed m31000| Mon Oct 15 21:26:38 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31001 heartbeat failed, retrying m31000| Mon Oct 15 21:26:38 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31001 is down (or slow to respond): m31000| Mon Oct 15 21:26:38 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31001 is now in state DOWN m31000| Mon Oct 15 21:26:38 [rsMgr] can't see a majority of the set, relinquishing primary m31000| Mon Oct 15 21:26:38 [rsMgr] replSet relinquishing primary state m31000| Mon Oct 15 21:26:38 [rsMgr] replSet SECONDARY m31000| Mon Oct 15 21:26:38 [rsMgr] replSet closing client sockets after relinquishing primary Mon Oct 15 21:26:38 Socket recv() errno:104 Connection reset by peer 127.0.0.1:31000 Mon Oct 15 21:26:38 SocketException: remote: 127.0.0.1:31000 error: 9001 socket exception [1] server [127.0.0.1:31000] Mon Oct 15 21:26:38 DBClientCursor::init call() failed Mon Oct 15 21:26:38 query failed : admin.$cmd { replSetGetStatus: 1.0 } to: 127.0.0.1:31000 Mon Oct 15 21:26:38 Error: error doing query: failed src/mongo/shell/collection.js:155 failed to load: /mnt/data/slaves/Linux_64bit_Legacy/mongo/jstests/replsets/stepdown2.js m31000| Mon Oct 15 21:26:38 [conn10] end connection 127.0.0.1:58149 (1 connection now open) m31000| Mon Oct 15 21:26:38 got signal 15 (Terminated), will terminate after current cmd ends m31000| Mon Oct 15 21:26:38 [interruptThread] now exiting m31000| Mon Oct 15 21:26:38 dbexit: m31000| Mon Oct 15 21:26:38 [interruptThread] shutdown: going to close listening sockets... m31000| Mon Oct 15 21:26:38 [interruptThread] closing listening socket: 11 m31000| Mon Oct 15 21:26:38 [interruptThread] closing listening socket: 12 m31000| Mon Oct 15 21:26:38 [interruptThread] closing listening socket: 13 m31000| Mon Oct 15 21:26:38 [interruptThread] removing socket file: /tmp/mongodb-31000.sock m31000| Mon Oct 15 21:26:38 [interruptThread] shutdown: going to flush diaglog... m31000| Mon Oct 15 21:26:38 [interruptThread] shutdown: going to close sockets... m31000| Mon Oct 15 21:26:38 [interruptThread] shutdown: waiting for fs preallocator... m31000| Mon Oct 15 21:26:38 [interruptThread] shutdown: lock for final Mon Oct 15 21:26:39 got signal 15 (Terminated), will terminate after current cmd ends Mon Oct 15 21:26:39 [interruptThread] now exiting Mon Oct 15 21:26:39 dbexit: Mon Oct 15 21:26:39 [interruptThread] shutdown: going to close listening sockets... Mon Oct 15 21:26:39 [interruptThread] closing listening socket: 7 Mon Oct 15 21:26:39 [interruptThread] closing listening socket: 8 Mon Oct 15 21:26:39 [interruptThread] closing listening socket: 9 Mon Oct 15 21:26:39 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Mon Oct 15 21:26:39 [interruptThread] shutdown: going to flush diaglog... Mon Oct 15 21:26:39 [interruptThread] shutdown: going to close sockets... Mon Oct 15 21:26:39 [interruptThread] shutdown: waiting for fs preallocator... Mon Oct 15 21:26:39 [interruptThread] shutdown: lock for final commit... Mon Oct 15 21:26:39 [interruptThread] shutdown: final commit... Mon Oct 15 21:26:39 [interruptThread] shutdown: closing all files... Mon Oct 15 21:26:39 [interruptThread] closeAllFiles() finished Mon Oct 15 21:26:39 [interruptThread] journalCleanup... Mon Oct 15 21:26:39 [interruptThread] removeJournalFiles Mon Oct 15 21:26:39 [interruptThread] shutdown: removing fs lock... Mon Oct 15 21:26:39 dbexit: really exiting now