Test : read_pref_rs_client.js ... Command : /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongo --port 27999 --nodb /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js --eval TestData = new Object();TestData.testPath = "/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js";TestData.testFile = "read_pref_rs_client.js";TestData.testName = "read_pref_rs_client";TestData.noJournal = false;TestData.noJournalPrealloc = false;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null; Date : Fri Mar 29 08:09:45 2013 Fri Mar 29 08:09:45 [conn79] end connection 127.0.0.1:40838 (0 connections now open) MongoDB shell version: 2.2.4-rc1-pre- null ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001 ] 31000 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "basic", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "basic" } } ReplSetTest Starting.... Resetting db path '/data/db/basic-0' Fri Mar 29 08:09:45 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31000 --noprealloc --smallfiles --rest --replSet basic --dbpath /data/db/basic-0 m31000| note: noprealloc may hurt performance in many applications m31000| Fri Mar 29 08:09:45 [initandlisten] MongoDB starting : pid=20173 port=31000 dbpath=/data/db/basic-0 64-bit host=ip-10-2-11-210 m31000| Fri Mar 29 08:09:45 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31000| Fri Mar 29 08:09:45 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31000| Fri Mar 29 08:09:45 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31000| Fri Mar 29 08:09:45 [initandlisten] options: { dbpath: "/data/db/basic-0", noprealloc: true, oplogSize: 1, port: 31000, replSet: "basic", rest: true, smallfiles: true } m31000| Fri Mar 29 08:09:45 [initandlisten] journal dir=/data/db/basic-0/journal m31000| Fri Mar 29 08:09:45 [initandlisten] recover : no journal files present, no recovery needed m31000| Fri Mar 29 08:09:46 [initandlisten] waiting for connections on port 31000 m31000| Fri Mar 29 08:09:46 [websvr] admin web console waiting for connections on port 32000 m31000| Fri Mar 29 08:09:46 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Mar 29 08:09:46 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Fri Mar 29 08:09:46 [initandlisten] connection accepted from 127.0.0.1:40030 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001 ] 31001 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "basic", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "basic" } } ReplSetTest Starting.... Resetting db path '/data/db/basic-1' Fri Mar 29 08:09:46 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31001 --noprealloc --smallfiles --rest --replSet basic --dbpath /data/db/basic-1 m31001| note: noprealloc may hurt performance in many applications m31001| Fri Mar 29 08:09:46 [initandlisten] MongoDB starting : pid=20221 port=31001 dbpath=/data/db/basic-1 64-bit host=ip-10-2-11-210 m31001| Fri Mar 29 08:09:46 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31001| Fri Mar 29 08:09:46 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31001| Fri Mar 29 08:09:46 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31001| Fri Mar 29 08:09:46 [initandlisten] options: { dbpath: "/data/db/basic-1", noprealloc: true, oplogSize: 1, port: 31001, replSet: "basic", rest: true, smallfiles: true } m31001| Fri Mar 29 08:09:46 [initandlisten] journal dir=/data/db/basic-1/journal m31001| Fri Mar 29 08:09:46 [initandlisten] recover : no journal files present, no recovery needed m31001| Fri Mar 29 08:09:46 [initandlisten] waiting for connections on port 31001 m31001| Fri Mar 29 08:09:46 [websvr] admin web console waiting for connections on port 32001 m31001| Fri Mar 29 08:09:46 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Fri Mar 29 08:09:46 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Fri Mar 29 08:09:46 [initandlisten] connection accepted from 127.0.0.1:37674 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001 ] { "replSetInitiate" : { "_id" : "basic", "members" : [ { "_id" : 0, "host" : "ip-10-2-11-210:31000" }, { "_id" : 1, "host" : "ip-10-2-11-210:31001" } ] } } m31000| Fri Mar 29 08:09:46 [conn1] replSet replSetInitiate admin command received from client m31000| Fri Mar 29 08:09:46 [conn1] replSet replSetInitiate config object parses ok, 2 members specified m31001| Fri Mar 29 08:09:46 [initandlisten] connection accepted from 10.2.11.210:59652 #2 (2 connections now open) m31000| Fri Mar 29 08:09:46 [conn1] replSet replSetInitiate all members seem up m31000| Fri Mar 29 08:09:46 [conn1] ****** m31000| Fri Mar 29 08:09:46 [conn1] creating replication oplog of size: 1MB... m31000| Fri Mar 29 08:09:46 [FileAllocator] allocating new datafile /data/db/basic-0/local.ns, filling with zeroes... m31000| Fri Mar 29 08:09:46 [FileAllocator] creating directory /data/db/basic-0/_tmp m31000| Fri Mar 29 08:09:46 [FileAllocator] done allocating datafile /data/db/basic-0/local.ns, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:09:46 [FileAllocator] allocating new datafile /data/db/basic-0/local.0, filling with zeroes... m31000| Fri Mar 29 08:09:46 [FileAllocator] done allocating datafile /data/db/basic-0/local.0, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:09:46 [conn1] ****** m31000| Fri Mar 29 08:09:46 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Mar 29 08:09:46 [conn1] replSet saveConfigLocally done m31000| Fri Mar 29 08:09:46 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31000| Fri Mar 29 08:09:56 [rsStart] replSet I am ip-10-2-11-210:31000 m31000| Fri Mar 29 08:09:56 [rsStart] replSet STARTUP2 m31000| Fri Mar 29 08:09:56 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31000| Fri Mar 29 08:09:56 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31001| Fri Mar 29 08:09:56 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:09:56 [initandlisten] connection accepted from 10.2.11.210:58191 #2 (2 connections now open) m31001| Fri Mar 29 08:09:56 [rsStart] replSet I am ip-10-2-11-210:31001 m31001| Fri Mar 29 08:09:56 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Fri Mar 29 08:09:56 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Fri Mar 29 08:09:56 [FileAllocator] allocating new datafile /data/db/basic-1/local.ns, filling with zeroes... m31001| Fri Mar 29 08:09:56 [FileAllocator] creating directory /data/db/basic-1/_tmp m31001| Fri Mar 29 08:09:56 [FileAllocator] done allocating datafile /data/db/basic-1/local.ns, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:09:56 [FileAllocator] allocating new datafile /data/db/basic-1/local.0, filling with zeroes... m31001| Fri Mar 29 08:09:56 [FileAllocator] done allocating datafile /data/db/basic-1/local.0, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:09:56 [rsStart] replSet saveConfigLocally done m31001| Fri Mar 29 08:09:56 [rsStart] replSet STARTUP2 m31001| Fri Mar 29 08:09:56 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31001| Fri Mar 29 08:09:56 [rsSync] ****** m31001| Fri Mar 29 08:09:56 [rsSync] creating replication oplog of size: 1MB... m31001| Fri Mar 29 08:09:56 [rsSync] ****** m31001| Fri Mar 29 08:09:56 [rsSync] replSet initial sync pending m31001| Fri Mar 29 08:09:56 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Fri Mar 29 08:09:57 [rsSync] replSet SECONDARY m31000| Fri Mar 29 08:09:58 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:09:58 [rsMgr] not electing self, ip-10-2-11-210:31001 would veto m31001| Fri Mar 29 08:09:58 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31001| Fri Mar 29 08:09:58 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31000| Fri Mar 29 08:10:04 [rsMgr] replSet info electSelf 0 m31001| Fri Mar 29 08:10:04 [conn2] replSet RECOVERING m31001| Fri Mar 29 08:10:04 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31000| Fri Mar 29 08:10:04 [rsMgr] replSet PRIMARY m31001| Fri Mar 29 08:10:04 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31000| Fri Mar 29 08:10:06 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state RECOVERING m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync pending m31001| Fri Mar 29 08:10:12 [rsSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:12 [initandlisten] connection accepted from 10.2.11.210:58192 #3 (3 connections now open) m31001| Fri Mar 29 08:10:12 [rsSync] build index local.me { _id: 1 } m31001| Fri Mar 29 08:10:12 [rsSync] build index done. scanned 0 total records. 0.001 secs m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync drop all databases m31001| Fri Mar 29 08:10:12 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync clone all databases m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync data copy, starting syncup m31001| Fri Mar 29 08:10:12 [rsSync] oplog sync 1 of 3 m31001| Fri Mar 29 08:10:12 [rsSync] oplog sync 2 of 3 m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync building indexes m31001| Fri Mar 29 08:10:12 [rsSync] oplog sync 3 of 3 m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync finishing up m31001| Fri Mar 29 08:10:12 [rsSync] replSet set minValid=51554c4a:1 m31001| Fri Mar 29 08:10:12 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Fri Mar 29 08:10:12 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Fri Mar 29 08:10:12 [rsSync] replSet initial sync done m31000| Fri Mar 29 08:10:12 [conn3] end connection 10.2.11.210:58192 (2 connections now open) m31001| Fri Mar 29 08:10:13 [rsBackgroundSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:13 [initandlisten] connection accepted from 10.2.11.210:58193 #4 (3 connections now open) m31001| Fri Mar 29 08:10:13 [rsSyncNotifier] replset setting oplog notifier to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:13 [initandlisten] connection accepted from 10.2.11.210:58194 #5 (4 connections now open) m31000| Fri Mar 29 08:10:14 [slaveTracking] build index local.slaves { _id: 1 } m31000| Fri Mar 29 08:10:14 [slaveTracking] build index done. scanned 0 total records. 0 secs m31001| Fri Mar 29 08:10:14 [rsSync] replSet SECONDARY Fri Mar 29 08:10:14 starting new replica set monitor for replica set basic with seed of ip-10-2-11-210:31000,ip-10-2-11-210:31001 m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58195 #6 (5 connections now open) Fri Mar 29 08:10:14 successfully connected to seed ip-10-2-11-210:31000 for replica set basic Fri Mar 29 08:10:14 changing hosts to { 0: "ip-10-2-11-210:31000", 1: "ip-10-2-11-210:31001" } from basic/ Fri Mar 29 08:10:14 trying to add new host ip-10-2-11-210:31000 to replica set basic m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58196 #7 (6 connections now open) Fri Mar 29 08:10:14 successfully connected to new host ip-10-2-11-210:31000 in replica set basic Fri Mar 29 08:10:14 trying to add new host ip-10-2-11-210:31001 to replica set basic m31001| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:59659 #3 (3 connections now open) Fri Mar 29 08:10:14 successfully connected to new host ip-10-2-11-210:31001 in replica set basic m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58198 #8 (7 connections now open) m31000| Fri Mar 29 08:10:14 [conn6] end connection 10.2.11.210:58195 (6 connections now open) Fri Mar 29 08:10:14 Primary for replica set basic changed to ip-10-2-11-210:31000 m31001| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:59661 #4 (4 connections now open) Fri Mar 29 08:10:14 replica set monitor for replica set basic started, address is basic/ip-10-2-11-210:31000,ip-10-2-11-210:31001 Fri Mar 29 08:10:14 [ReplicaSetMonitorWatcher] starting m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58200 #9 (7 connections now open) m31001| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:59663 #5 (5 connections now open) m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58202 #10 (8 connections now open) m31001| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:59665 #6 (6 connections now open) m31000| Fri Mar 29 08:10:14 [initandlisten] connection accepted from 10.2.11.210:58204 #11 (9 connections now open) ReplSetTest n: 0 ports: [ 31000, 31001 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** m31000| Fri Mar 29 08:10:14 got signal 15 (Terminated), will terminate after current cmd ends m31000| Fri Mar 29 08:10:14 [interruptThread] now exiting m31000| Fri Mar 29 08:10:14 dbexit: m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: going to close listening sockets... m31000| Fri Mar 29 08:10:14 [interruptThread] closing listening socket: 10 m31000| Fri Mar 29 08:10:14 [interruptThread] closing listening socket: 11 m31000| Fri Mar 29 08:10:14 [interruptThread] closing listening socket: 12 m31000| Fri Mar 29 08:10:14 [interruptThread] removing socket file: /tmp/mongodb-31000.sock m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: going to flush diaglog... m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: going to close sockets... m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: waiting for fs preallocator... m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: lock for final commit... m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: final commit... m31001| Fri Mar 29 08:10:14 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: ip-10-2-11-210:31000 m31001| Fri Mar 29 08:10:14 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-10-2-11-210:31000 m31001| Fri Mar 29 08:10:14 [conn2] end connection 10.2.11.210:59652 (5 connections now open) m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: closing all files... m31000| Fri Mar 29 08:10:14 [interruptThread] closeAllFiles() finished m31000| Fri Mar 29 08:10:14 [interruptThread] journalCleanup... m31000| Fri Mar 29 08:10:14 [interruptThread] removeJournalFiles m31000| Fri Mar 29 08:10:14 [interruptThread] shutdown: removing fs lock... m31000| Fri Mar 29 08:10:14 dbexit: really exiting now Fri Mar 29 08:10:15 shell: stopped mongo program on port 31000 ReplSetTest n: 1 ports: [ 31000, 31001 ] 31001 number ReplSetTest stop *** Shutting down mongod in port 31001 *** m31001| Fri Mar 29 08:10:15 got signal 15 (Terminated), will terminate after current cmd ends m31001| Fri Mar 29 08:10:15 [interruptThread] now exiting m31001| Fri Mar 29 08:10:15 dbexit: m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: going to close listening sockets... m31001| Fri Mar 29 08:10:15 [interruptThread] closing listening socket: 13 m31001| Fri Mar 29 08:10:15 [interruptThread] closing listening socket: 14 m31001| Fri Mar 29 08:10:15 [interruptThread] closing listening socket: 15 m31001| Fri Mar 29 08:10:15 [interruptThread] removing socket file: /tmp/mongodb-31001.sock m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: going to flush diaglog... m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: going to close sockets... m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: waiting for fs preallocator... m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: lock for final commit... m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: final commit... m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: closing all files... m31001| Fri Mar 29 08:10:15 [interruptThread] closeAllFiles() finished m31001| Fri Mar 29 08:10:15 [interruptThread] journalCleanup... m31001| Fri Mar 29 08:10:15 [interruptThread] removeJournalFiles m31001| Fri Mar 29 08:10:15 [interruptThread] shutdown: removing fs lock... m31001| Fri Mar 29 08:10:15 dbexit: really exiting now Fri Mar 29 08:10:16 shell: stopped mongo program on port 31001 ReplSetTest stopSet deleting all dbpaths ReplSetTest stopSet *** Shut down repl set - test worked **** ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priOnlyNoPri", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "priOnlyNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priOnlyNoPri-0' Fri Mar 29 08:10:16 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31000 --noprealloc --smallfiles --rest --replSet priOnlyNoPri --dbpath /data/db/priOnlyNoPri-0 m31000| note: noprealloc may hurt performance in many applications m31000| Fri Mar 29 08:10:16 [initandlisten] MongoDB starting : pid=20318 port=31000 dbpath=/data/db/priOnlyNoPri-0 64-bit host=ip-10-2-11-210 m31000| Fri Mar 29 08:10:16 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31000| Fri Mar 29 08:10:16 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31000| Fri Mar 29 08:10:16 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31000| Fri Mar 29 08:10:16 [initandlisten] options: { dbpath: "/data/db/priOnlyNoPri-0", noprealloc: true, oplogSize: 1, port: 31000, replSet: "priOnlyNoPri", rest: true, smallfiles: true } m31000| Fri Mar 29 08:10:16 [initandlisten] journal dir=/data/db/priOnlyNoPri-0/journal m31000| Fri Mar 29 08:10:16 [initandlisten] recover : no journal files present, no recovery needed m31000| Fri Mar 29 08:10:17 [initandlisten] waiting for connections on port 31000 m31000| Fri Mar 29 08:10:17 [websvr] admin web console waiting for connections on port 32000 m31000| Fri Mar 29 08:10:17 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Mar 29 08:10:17 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Fri Mar 29 08:10:17 [initandlisten] connection accepted from 127.0.0.1:40051 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priOnlyNoPri", "dbpath" : "$set-$node", "arbiter" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "priOnlyNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priOnlyNoPri-1' Fri Mar 29 08:10:17 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31001 --noprealloc --smallfiles --rest --replSet priOnlyNoPri --dbpath /data/db/priOnlyNoPri-1 m31001| note: noprealloc may hurt performance in many applications m31001| Fri Mar 29 08:10:17 [initandlisten] MongoDB starting : pid=20366 port=31001 dbpath=/data/db/priOnlyNoPri-1 64-bit host=ip-10-2-11-210 m31001| Fri Mar 29 08:10:17 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31001| Fri Mar 29 08:10:17 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31001| Fri Mar 29 08:10:17 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31001| Fri Mar 29 08:10:17 [initandlisten] options: { dbpath: "/data/db/priOnlyNoPri-1", noprealloc: true, oplogSize: 1, port: 31001, replSet: "priOnlyNoPri", rest: true, smallfiles: true } m31001| Fri Mar 29 08:10:17 [initandlisten] journal dir=/data/db/priOnlyNoPri-1/journal m31001| Fri Mar 29 08:10:17 [initandlisten] recover : no journal files present, no recovery needed m31001| Fri Mar 29 08:10:17 [initandlisten] waiting for connections on port 31001 m31001| Fri Mar 29 08:10:17 [websvr] admin web console waiting for connections on port 32001 m31001| Fri Mar 29 08:10:17 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Fri Mar 29 08:10:17 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Fri Mar 29 08:10:17 [initandlisten] connection accepted from 127.0.0.1:37695 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priOnlyNoPri", "dbpath" : "$set-$node", "arbiter" : true, "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "priOnlyNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priOnlyNoPri-2' Fri Mar 29 08:10:17 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31002 --noprealloc --smallfiles --rest --replSet priOnlyNoPri --dbpath /data/db/priOnlyNoPri-2 m31002| note: noprealloc may hurt performance in many applications m31002| Fri Mar 29 08:10:17 [initandlisten] MongoDB starting : pid=20414 port=31002 dbpath=/data/db/priOnlyNoPri-2 64-bit host=ip-10-2-11-210 m31002| Fri Mar 29 08:10:17 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31002| Fri Mar 29 08:10:17 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31002| Fri Mar 29 08:10:17 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31002| Fri Mar 29 08:10:17 [initandlisten] options: { dbpath: "/data/db/priOnlyNoPri-2", noprealloc: true, oplogSize: 1, port: 31002, replSet: "priOnlyNoPri", rest: true, smallfiles: true } m31002| Fri Mar 29 08:10:17 [initandlisten] journal dir=/data/db/priOnlyNoPri-2/journal m31002| Fri Mar 29 08:10:17 [initandlisten] recover : no journal files present, no recovery needed m31002| Fri Mar 29 08:10:17 [initandlisten] waiting for connections on port 31002 m31002| Fri Mar 29 08:10:17 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Fri Mar 29 08:10:17 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Fri Mar 29 08:10:17 [websvr] admin web console waiting for connections on port 32002 m31002| Fri Mar 29 08:10:17 [initandlisten] connection accepted from 127.0.0.1:62634 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001, connection to ip-10-2-11-210:31002 ] { "replSetInitiate" : { "_id" : "priOnlyNoPri", "members" : [ { "_id" : 0, "host" : "ip-10-2-11-210:31000" }, { "_id" : 1, "host" : "ip-10-2-11-210:31001", "arbiterOnly" : true }, { "_id" : 2, "host" : "ip-10-2-11-210:31002", "arbiterOnly" : true } ] } } m31000| Fri Mar 29 08:10:17 [conn1] replSet replSetInitiate admin command received from client m31000| Fri Mar 29 08:10:17 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Fri Mar 29 08:10:17 [initandlisten] connection accepted from 10.2.11.210:59676 #2 (2 connections now open) m31002| Fri Mar 29 08:10:17 [initandlisten] connection accepted from 10.2.11.210:47830 #2 (2 connections now open) m31000| Fri Mar 29 08:10:17 [conn1] replSet replSetInitiate all members seem up m31000| Fri Mar 29 08:10:17 [conn1] ****** m31000| Fri Mar 29 08:10:17 [conn1] creating replication oplog of size: 1MB... m31000| Fri Mar 29 08:10:17 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-0/local.ns, filling with zeroes... m31000| Fri Mar 29 08:10:17 [FileAllocator] creating directory /data/db/priOnlyNoPri-0/_tmp m31000| Fri Mar 29 08:10:17 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-0/local.ns, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:10:17 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-0/local.0, filling with zeroes... m31000| Fri Mar 29 08:10:17 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-0/local.0, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:10:17 [conn1] ****** m31000| Fri Mar 29 08:10:17 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Mar 29 08:10:17 [conn1] replSet saveConfigLocally done m31000| Fri Mar 29 08:10:17 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } Fri Mar 29 08:10:24 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:10:24 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:24 [initandlisten] connection accepted from 10.2.11.210:58216 #2 (2 connections now open) Fri Mar 29 08:10:24 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:10:24 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544624680), ok: 1.0 } Fri Mar 29 08:10:24 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:10:25 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544625680), ok: 1.0 } Fri Mar 29 08:10:25 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31001 m31001| Fri Mar 29 08:10:25 [initandlisten] connection accepted from 10.2.11.210:59679 #3 (3 connections now open) Fri Mar 29 08:10:25 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31001 ok Fri Mar 29 08:10:25 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544625681), ok: 1.0 } Fri Mar 29 08:10:26 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:10:26 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks m31000| Fri Mar 29 08:10:27 [rsStart] replSet I am ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:27 [rsStart] replSet STARTUP2 m31000| Fri Mar 29 08:10:27 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31000| Fri Mar 29 08:10:27 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31001| Fri Mar 29 08:10:27 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:27 [initandlisten] connection accepted from 10.2.11.210:58218 #3 (3 connections now open) m31001| Fri Mar 29 08:10:27 [rsStart] replSet I am ip-10-2-11-210:31001 m31001| Fri Mar 29 08:10:27 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Fri Mar 29 08:10:27 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Fri Mar 29 08:10:27 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-1/local.ns, filling with zeroes... m31001| Fri Mar 29 08:10:27 [FileAllocator] creating directory /data/db/priOnlyNoPri-1/_tmp m31001| Fri Mar 29 08:10:27 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-1/local.ns, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:10:27 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-1/local.0, filling with zeroes... m31001| Fri Mar 29 08:10:27 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-1/local.0, size: 16MB, took 0.001 secs m31001| Fri Mar 29 08:10:27 [rsStart] replSet saveConfigLocally done m31001| Fri Mar 29 08:10:27 [rsStart] replSet STARTUP2 m31002| Fri Mar 29 08:10:27 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:10:27 [initandlisten] connection accepted from 10.2.11.210:58219 #4 (4 connections now open) m31002| Fri Mar 29 08:10:27 [rsStart] replSet I am ip-10-2-11-210:31002 m31002| Fri Mar 29 08:10:27 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Fri Mar 29 08:10:27 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Fri Mar 29 08:10:27 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-2/local.ns, filling with zeroes... m31002| Fri Mar 29 08:10:27 [FileAllocator] creating directory /data/db/priOnlyNoPri-2/_tmp m31002| Fri Mar 29 08:10:27 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-2/local.ns, size: 16MB, took 0.001 secs m31002| Fri Mar 29 08:10:27 [FileAllocator] allocating new datafile /data/db/priOnlyNoPri-2/local.0, filling with zeroes... m31002| Fri Mar 29 08:10:27 [FileAllocator] done allocating datafile /data/db/priOnlyNoPri-2/local.0, size: 16MB, took 0.001 secs m31002| Fri Mar 29 08:10:27 [rsStart] replSet saveConfigLocally done m31002| Fri Mar 29 08:10:27 [rsStart] replSet STARTUP2 m31000| Fri Mar 29 08:10:28 [rsSync] replSet SECONDARY m31000| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 m31000| Fri Mar 29 08:10:29 [rsMgr] not electing self, ip-10-2-11-210:31002 would veto m31000| Fri Mar 29 08:10:29 [rsMgr] not electing self, ip-10-2-11-210:31002 would veto m31002| Fri Mar 29 08:10:29 [initandlisten] connection accepted from 10.2.11.210:47835 #3 (3 connections now open) m31001| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31001| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31001| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31001| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 m31001| Fri Mar 29 08:10:29 [initandlisten] connection accepted from 10.2.11.210:59683 #4 (4 connections now open) m31002| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31002| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31002| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31002| Fri Mar 29 08:10:29 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:10:35 [rsMgr] replSet info electSelf 0 m31002| Fri Mar 29 08:10:35 [conn2] replSet RECOVERING m31002| Fri Mar 29 08:10:35 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31001| Fri Mar 29 08:10:35 [conn2] replSet RECOVERING m31001| Fri Mar 29 08:10:35 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31000| Fri Mar 29 08:10:35 [rsMgr] replSet PRIMARY m31001| Fri Mar 29 08:10:35 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state RECOVERING m31001| Fri Mar 29 08:10:35 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31002| Fri Mar 29 08:10:35 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state ARBITER m31002| Fri Mar 29 08:10:35 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY Fri Mar 29 08:10:35 starting new replica set monitor for replica set priOnlyNoPri with seed of ip-10-2-11-210:31000,ip-10-2-11-210:31001,ip-10-2-11-210:31002 m31000| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:58222 #5 (5 connections now open) Fri Mar 29 08:10:35 successfully connected to seed ip-10-2-11-210:31000 for replica set priOnlyNoPri Fri Mar 29 08:10:35 changing hosts to { 0: "ip-10-2-11-210:31000" } from priOnlyNoPri/ Fri Mar 29 08:10:35 trying to add new host ip-10-2-11-210:31000 to replica set priOnlyNoPri m31000| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:58223 #6 (6 connections now open) Fri Mar 29 08:10:35 successfully connected to new host ip-10-2-11-210:31000 in replica set priOnlyNoPri Fri Mar 29 08:10:35 DBClientCursor::init call() failed m31001| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:59686 #5 (5 connections now open) Fri Mar 29 08:10:35 successfully connected to seed ip-10-2-11-210:31001 for replica set priOnlyNoPri m31000| Fri Mar 29 08:10:35 [conn5] end connection 10.2.11.210:58222 (5 connections now open) Fri Mar 29 08:10:35 DBClientCursor::init call() failed m31001| Fri Mar 29 08:10:35 [conn5] end connection 10.2.11.210:59686 (4 connections now open) m31002| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:47840 #4 (4 connections now open) Fri Mar 29 08:10:35 successfully connected to seed ip-10-2-11-210:31002 for replica set priOnlyNoPri m31002| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:47841 #5 (5 connections now open) m31002| Fri Mar 29 08:10:35 [conn4] end connection 10.2.11.210:47840 (4 connections now open) m31000| Fri Mar 29 08:10:35 [initandlisten] connection accepted from 10.2.11.210:58227 #7 (6 connections now open) Fri Mar 29 08:10:35 Primary for replica set priOnlyNoPri changed to ip-10-2-11-210:31000 Fri Mar 29 08:10:35 replica set monitor for replica set priOnlyNoPri started, address is priOnlyNoPri/ip-10-2-11-210:31000 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** m31000| Fri Mar 29 08:10:35 got signal 15 (Terminated), will terminate after current cmd ends m31000| Fri Mar 29 08:10:35 [interruptThread] now exiting m31000| Fri Mar 29 08:10:35 dbexit: m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: going to close listening sockets... m31000| Fri Mar 29 08:10:35 [interruptThread] closing listening socket: 21 m31000| Fri Mar 29 08:10:35 [interruptThread] closing listening socket: 22 m31000| Fri Mar 29 08:10:35 [interruptThread] closing listening socket: 23 m31000| Fri Mar 29 08:10:35 [interruptThread] removing socket file: /tmp/mongodb-31000.sock m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: going to flush diaglog... m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: going to close sockets... m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: waiting for fs preallocator... m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: lock for final commit... m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: final commit... m31002| Fri Mar 29 08:10:35 [conn2] end connection 10.2.11.210:47830 (3 connections now open) m31001| Fri Mar 29 08:10:35 [conn2] end connection 10.2.11.210:59676 (3 connections now open) m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: closing all files... m31000| Fri Mar 29 08:10:35 [interruptThread] closeAllFiles() finished m31000| Fri Mar 29 08:10:35 [interruptThread] journalCleanup... m31000| Fri Mar 29 08:10:35 [interruptThread] removeJournalFiles m31000| Fri Mar 29 08:10:35 [interruptThread] shutdown: removing fs lock... m31000| Fri Mar 29 08:10:35 dbexit: really exiting now Fri Mar 29 08:10:36 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:10:36 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priOnlyNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001", "ip-10-2-11-210:31002" ], primary: "ip-10-2-11-210:31000", arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544636682), ok: 1.0 } Fri Mar 29 08:10:36 shell: stopped mongo program on port 31000 Fri Mar 29 08:10:36 DBClientCursor::init call() failed m31001| Fri Mar 29 08:10:37 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state ARBITER m31001| Fri Mar 29 08:10:37 [rsHealthPoll] DBClientCursor::init call() failed m31001| Fri Mar 29 08:10:37 [rsHealthPoll] replSet info ip-10-2-11-210:31000 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31000 ns: admin.$cmd query: { replSetHeartbeat: "priOnlyNoPri", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31001", $auth: {} } m31001| Fri Mar 29 08:10:37 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state DOWN Fri Mar 29 08:10:37 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:37 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:37 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:37 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priOnlyNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001", "ip-10-2-11-210:31002" ], arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544637683), ok: 1.0 } m31002| Fri Mar 29 08:10:37 [rsHealthPoll] DBClientCursor::init call() failed m31002| Fri Mar 29 08:10:37 [rsHealthPoll] replSet info ip-10-2-11-210:31000 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31000 ns: admin.$cmd query: { replSetHeartbeat: "priOnlyNoPri", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31002", $auth: {} } m31002| Fri Mar 29 08:10:37 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state DOWN Fri Mar 29 08:10:37 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:37 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:37 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:38 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:10:38 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks Fri Mar 29 08:10:38 warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:10:39 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:39 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:39 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:39 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [FAILED_STATE] for ip-10-2-11-210:31000 Fri Mar 29 08:10:40 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:10:40 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks Fri Mar 29 08:10:40 warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:10:41 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:41 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:41 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:42 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:43 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31001| Fri Mar 29 08:10:43 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:43 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:43 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:43 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:43 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:43 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:44 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:45 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:45 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:45 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:45 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:45 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:46 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:47 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:47 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:47 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:47 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:47 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:48 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:49 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:49 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:49 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:49 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:49 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:50 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:50 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:50 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priOnlyNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001", "ip-10-2-11-210:31002" ], arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544650685), ok: 1.0 } Fri Mar 29 08:10:50 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:51 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [FAILED_STATE] for ip-10-2-11-210:31000 Fri Mar 29 08:10:51 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priOnlyNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001", "ip-10-2-11-210:31002" ], arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544651685), ok: 1.0 } m31002| Fri Mar 29 08:10:51 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:51 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:51 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:51 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 Fri Mar 29 08:10:52 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:10:52 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks Fri Mar 29 08:10:52 warning: No primary detected for set priOnlyNoPri m31001| Fri Mar 29 08:10:53 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:53 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:10:53 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:53 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [CONNECT_ERROR] for ip-10-2-11-210:31000 m31002| Fri Mar 29 08:10:53 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:53 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [FAILED_STATE] for ip-10-2-11-210:31000 Fri Mar 29 08:10:54 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:10:54 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks Fri Mar 29 08:10:54 warning: No primary detected for set priOnlyNoPri ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** Fri Mar 29 08:10:54 No db started on port: 31000 Fri Mar 29 08:10:54 shell: stopped mongo program on port 31000 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number ReplSetTest stop *** Shutting down mongod in port 31001 *** m31001| Fri Mar 29 08:10:54 got signal 15 (Terminated), will terminate after current cmd ends m31001| Fri Mar 29 08:10:54 [interruptThread] now exiting m31001| Fri Mar 29 08:10:54 dbexit: m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: going to close listening sockets... m31001| Fri Mar 29 08:10:54 [interruptThread] closing listening socket: 24 m31001| Fri Mar 29 08:10:54 [interruptThread] closing listening socket: 25 m31001| Fri Mar 29 08:10:54 [interruptThread] closing listening socket: 26 m31001| Fri Mar 29 08:10:54 [interruptThread] removing socket file: /tmp/mongodb-31001.sock m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: going to flush diaglog... m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: going to close sockets... m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: waiting for fs preallocator... m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: lock for final commit... m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: final commit... m31002| Fri Mar 29 08:10:54 [conn3] end connection 10.2.11.210:47835 (2 connections now open) m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: closing all files... m31001| Fri Mar 29 08:10:54 [interruptThread] closeAllFiles() finished m31001| Fri Mar 29 08:10:54 [interruptThread] journalCleanup... m31001| Fri Mar 29 08:10:54 [interruptThread] removeJournalFiles m31001| Fri Mar 29 08:10:54 [interruptThread] shutdown: removing fs lock... m31001| Fri Mar 29 08:10:54 dbexit: really exiting now m31002| Fri Mar 29 08:10:55 [rsHealthPoll] DBClientCursor::init call() failed m31002| Fri Mar 29 08:10:55 [rsHealthPoll] replSet info ip-10-2-11-210:31001 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31001 ns: admin.$cmd query: { replSetHeartbeat: "priOnlyNoPri", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31002", $auth: {} } m31002| Fri Mar 29 08:10:55 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state DOWN m31002| Fri Mar 29 08:10:55 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31000: couldn't connect to server ip-10-2-11-210:31000 Fri Mar 29 08:10:55 shell: stopped mongo program on port 31001 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number ReplSetTest stop *** Shutting down mongod in port 31002 *** m31002| Fri Mar 29 08:10:55 got signal 15 (Terminated), will terminate after current cmd ends m31002| Fri Mar 29 08:10:55 [interruptThread] now exiting m31002| Fri Mar 29 08:10:55 dbexit: m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: going to close listening sockets... m31002| Fri Mar 29 08:10:55 [interruptThread] closing listening socket: 27 m31002| Fri Mar 29 08:10:55 [interruptThread] closing listening socket: 28 m31002| Fri Mar 29 08:10:55 [interruptThread] closing listening socket: 29 m31002| Fri Mar 29 08:10:55 [interruptThread] removing socket file: /tmp/mongodb-31002.sock m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: going to flush diaglog... m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: going to close sockets... m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: waiting for fs preallocator... m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: lock for final commit... m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: final commit... m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: closing all files... m31002| Fri Mar 29 08:10:55 [interruptThread] closeAllFiles() finished m31002| Fri Mar 29 08:10:55 [interruptThread] journalCleanup... m31002| Fri Mar 29 08:10:55 [interruptThread] removeJournalFiles m31002| Fri Mar 29 08:10:55 [interruptThread] shutdown: removing fs lock... m31002| Fri Mar 29 08:10:55 dbexit: really exiting now Fri Mar 29 08:10:56 shell: stopped mongo program on port 31002 ReplSetTest stopSet deleting all dbpaths ReplSetTest stopSet *** Shut down repl set - test worked **** ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "secOnlyNoSec", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "secOnlyNoSec" } } ReplSetTest Starting.... Resetting db path '/data/db/secOnlyNoSec-0' Fri Mar 29 08:10:56 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31000 --noprealloc --smallfiles --rest --replSet secOnlyNoSec --dbpath /data/db/secOnlyNoSec-0 m31000| note: noprealloc may hurt performance in many applications m31000| Fri Mar 29 08:10:57 [initandlisten] MongoDB starting : pid=20558 port=31000 dbpath=/data/db/secOnlyNoSec-0 64-bit host=ip-10-2-11-210 m31000| Fri Mar 29 08:10:57 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31000| Fri Mar 29 08:10:57 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31000| Fri Mar 29 08:10:57 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31000| Fri Mar 29 08:10:57 [initandlisten] options: { dbpath: "/data/db/secOnlyNoSec-0", noprealloc: true, oplogSize: 1, port: 31000, replSet: "secOnlyNoSec", rest: true, smallfiles: true } m31000| Fri Mar 29 08:10:57 [initandlisten] journal dir=/data/db/secOnlyNoSec-0/journal m31000| Fri Mar 29 08:10:57 [initandlisten] recover : no journal files present, no recovery needed m31000| Fri Mar 29 08:10:57 [initandlisten] preallocateIsFaster=true 2.6 m31000| Fri Mar 29 08:10:57 [initandlisten] waiting for connections on port 31000 m31000| Fri Mar 29 08:10:57 [websvr] admin web console waiting for connections on port 32000 m31000| Fri Mar 29 08:10:57 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Mar 29 08:10:57 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Fri Mar 29 08:10:57 [initandlisten] connection accepted from 127.0.0.1:40120 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "secOnlyNoSec", "dbpath" : "$set-$node", "arbiter" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "secOnlyNoSec" } } ReplSetTest Starting.... Resetting db path '/data/db/secOnlyNoSec-1' Fri Mar 29 08:10:57 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31001 --noprealloc --smallfiles --rest --replSet secOnlyNoSec --dbpath /data/db/secOnlyNoSec-1 m31001| note: noprealloc may hurt performance in many applications m31001| Fri Mar 29 08:10:57 [initandlisten] MongoDB starting : pid=20608 port=31001 dbpath=/data/db/secOnlyNoSec-1 64-bit host=ip-10-2-11-210 m31001| Fri Mar 29 08:10:57 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31001| Fri Mar 29 08:10:57 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31001| Fri Mar 29 08:10:57 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31001| Fri Mar 29 08:10:57 [initandlisten] options: { dbpath: "/data/db/secOnlyNoSec-1", noprealloc: true, oplogSize: 1, port: 31001, replSet: "secOnlyNoSec", rest: true, smallfiles: true } m31001| Fri Mar 29 08:10:57 [initandlisten] journal dir=/data/db/secOnlyNoSec-1/journal m31001| Fri Mar 29 08:10:57 [initandlisten] recover : no journal files present, no recovery needed m31001| Fri Mar 29 08:10:58 [initandlisten] waiting for connections on port 31001 m31001| Fri Mar 29 08:10:58 [websvr] admin web console waiting for connections on port 32001 m31001| Fri Mar 29 08:10:58 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Fri Mar 29 08:10:58 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Fri Mar 29 08:10:58 [initandlisten] connection accepted from 127.0.0.1:37764 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "secOnlyNoSec", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "secOnlyNoSec" } } ReplSetTest Starting.... Resetting db path '/data/db/secOnlyNoSec-2' Fri Mar 29 08:10:58 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31002 --noprealloc --smallfiles --rest --replSet secOnlyNoSec --dbpath /data/db/secOnlyNoSec-2 m31002| note: noprealloc may hurt performance in many applications m31002| Fri Mar 29 08:10:58 [initandlisten] MongoDB starting : pid=20656 port=31002 dbpath=/data/db/secOnlyNoSec-2 64-bit host=ip-10-2-11-210 m31002| Fri Mar 29 08:10:58 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31002| Fri Mar 29 08:10:58 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31002| Fri Mar 29 08:10:58 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31002| Fri Mar 29 08:10:58 [initandlisten] options: { dbpath: "/data/db/secOnlyNoSec-2", noprealloc: true, oplogSize: 1, port: 31002, replSet: "secOnlyNoSec", rest: true, smallfiles: true } m31002| Fri Mar 29 08:10:58 [initandlisten] journal dir=/data/db/secOnlyNoSec-2/journal m31002| Fri Mar 29 08:10:58 [initandlisten] recover : no journal files present, no recovery needed m31002| Fri Mar 29 08:10:58 [initandlisten] waiting for connections on port 31002 m31002| Fri Mar 29 08:10:58 [websvr] admin web console waiting for connections on port 32002 m31002| Fri Mar 29 08:10:58 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Fri Mar 29 08:10:58 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Fri Mar 29 08:10:58 [initandlisten] connection accepted from 127.0.0.1:62703 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001, connection to ip-10-2-11-210:31002 ] { "replSetInitiate" : { "_id" : "secOnlyNoSec", "members" : [ { "_id" : 0, "host" : "ip-10-2-11-210:31000" }, { "_id" : 1, "host" : "ip-10-2-11-210:31001", "arbiterOnly" : true }, { "_id" : 2, "host" : "ip-10-2-11-210:31002" } ] } } m31000| Fri Mar 29 08:10:58 [conn1] replSet replSetInitiate admin command received from client m31000| Fri Mar 29 08:10:58 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Fri Mar 29 08:10:58 [initandlisten] connection accepted from 10.2.11.210:59745 #2 (2 connections now open) m31002| Fri Mar 29 08:10:58 [initandlisten] connection accepted from 10.2.11.210:47899 #2 (2 connections now open) m31000| Fri Mar 29 08:10:58 [conn1] replSet replSetInitiate all members seem up m31000| Fri Mar 29 08:10:58 [conn1] ****** m31000| Fri Mar 29 08:10:58 [conn1] creating replication oplog of size: 1MB... m31000| Fri Mar 29 08:10:58 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-0/local.ns, filling with zeroes... m31000| Fri Mar 29 08:10:58 [FileAllocator] creating directory /data/db/secOnlyNoSec-0/_tmp m31000| Fri Mar 29 08:10:58 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-0/local.ns, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:10:58 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-0/local.0, filling with zeroes... m31000| Fri Mar 29 08:10:58 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-0/local.0, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:10:58 [conn1] ****** m31000| Fri Mar 29 08:10:58 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Mar 29 08:10:58 [conn1] replSet saveConfigLocally done m31000| Fri Mar 29 08:10:58 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } Fri Mar 29 08:11:04 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:04 [initandlisten] connection accepted from 10.2.11.210:58285 #2 (2 connections now open) Fri Mar 29 08:11:04 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:11:04 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544664688), ok: 1.0 } Fri Mar 29 08:11:04 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:11:05 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544665688), ok: 1.0 } Fri Mar 29 08:11:05 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31001 m31001| Fri Mar 29 08:11:05 [initandlisten] connection accepted from 10.2.11.210:59748 #3 (3 connections now open) Fri Mar 29 08:11:05 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31001 ok Fri Mar 29 08:11:05 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544665689), ok: 1.0 } Fri Mar 29 08:11:06 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:11:06 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks Fri Mar 29 08:11:06 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:06 [initandlisten] connection accepted from 10.2.11.210:58287 #3 (3 connections now open) Fri Mar 29 08:11:06 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:11:06 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544666690), ok: 1.0 } m31000| Fri Mar 29 08:11:07 [rsStart] replSet I am ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:07 [rsStart] replSet STARTUP2 m31000| Fri Mar 29 08:11:07 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up Fri Mar 29 08:11:07 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544667690), ok: 1.0 } m31001| Fri Mar 29 08:11:08 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:08 [initandlisten] connection accepted from 10.2.11.210:58288 #4 (4 connections now open) m31001| Fri Mar 29 08:11:08 [rsStart] replSet I am ip-10-2-11-210:31001 m31001| Fri Mar 29 08:11:08 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Fri Mar 29 08:11:08 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Fri Mar 29 08:11:08 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-1/local.ns, filling with zeroes... m31001| Fri Mar 29 08:11:08 [FileAllocator] creating directory /data/db/secOnlyNoSec-1/_tmp m31001| Fri Mar 29 08:11:08 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-1/local.ns, size: 16MB, took 0.012 secs m31001| Fri Mar 29 08:11:08 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-1/local.0, filling with zeroes... m31001| Fri Mar 29 08:11:08 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-1/local.0, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:11:08 [rsStart] replSet saveConfigLocally done m31001| Fri Mar 29 08:11:08 [rsStart] replSet STARTUP2 m31002| Fri Mar 29 08:11:08 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Mar 29 08:11:08 [rsSync] replSet SECONDARY Fri Mar 29 08:11:08 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:11:08 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks m31000| Fri Mar 29 08:11:09 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31000| Fri Mar 29 08:11:09 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:11:09 [rsMgr] not electing self, ip-10-2-11-210:31001 would veto m31000| Fri Mar 29 08:11:09 [rsMgr] not electing self, ip-10-2-11-210:31001 would veto m31001| Fri Mar 29 08:11:10 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31001| Fri Mar 29 08:11:10 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31002| Fri Mar 29 08:11:10 [initandlisten] connection accepted from 10.2.11.210:47904 #3 (3 connections now open) m31001| Fri Mar 29 08:11:10 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31000| Fri Mar 29 08:11:15 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31002| Fri Mar 29 08:11:18 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:58290 #5 (5 connections now open) m31002| Fri Mar 29 08:11:18 [rsStart] trying to contact ip-10-2-11-210:31001 m31001| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:59753 #4 (4 connections now open) m31002| Fri Mar 29 08:11:18 [rsStart] replSet I am ip-10-2-11-210:31002 m31002| Fri Mar 29 08:11:18 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Fri Mar 29 08:11:18 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-2/local.ns, filling with zeroes... m31002| Fri Mar 29 08:11:18 [FileAllocator] creating directory /data/db/secOnlyNoSec-2/_tmp m31002| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-2/local.ns, size: 16MB, took 0.001 secs m31002| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/secOnlyNoSec-2/local.0, filling with zeroes... m31002| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/secOnlyNoSec-2/local.0, size: 16MB, took 0.001 secs m31002| Fri Mar 29 08:11:18 [rsStart] replSet saveConfigLocally done m31002| Fri Mar 29 08:11:18 [rsStart] replSet STARTUP2 m31002| Fri Mar 29 08:11:18 [rsSync] ****** m31002| Fri Mar 29 08:11:18 [rsSync] creating replication oplog of size: 1MB... m31002| Fri Mar 29 08:11:18 [rsSync] ****** m31002| Fri Mar 29 08:11:18 [rsSync] replSet initial sync pending m31002| Fri Mar 29 08:11:18 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544678695), ok: 1.0 } Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544678696), ok: 1.0 } m31000| Fri Mar 29 08:11:19 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 m31000| Fri Mar 29 08:11:19 [rsMgr] not electing self, ip-10-2-11-210:31002 would veto Fri Mar 29 08:11:19 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544679696), ok: 1.0 } Fri Mar 29 08:11:19 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544679697), ok: 1.0 } m31001| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 Fri Mar 29 08:11:20 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:11:20 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks Fri Mar 29 08:11:20 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544680697), ok: 1.0 } m31001| Fri Mar 29 08:11:21 [conn2] end connection 10.2.11.210:59745 (3 connections now open) m31001| Fri Mar 29 08:11:21 [initandlisten] connection accepted from 10.2.11.210:59754 #5 (4 connections now open) Fri Mar 29 08:11:21 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544681698), ok: 1.0 } Fri Mar 29 08:11:22 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:11:22 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks m31000| Fri Mar 29 08:11:24 [initandlisten] connection accepted from 10.2.11.210:58293 #6 (6 connections now open) m31000| Fri Mar 29 08:11:24 [conn4] end connection 10.2.11.210:58288 (4 connections now open) m31000| Fri Mar 29 08:11:25 [rsMgr] replSet info electSelf 0 m31001| Fri Mar 29 08:11:25 [conn5] replSet RECOVERING m31001| Fri Mar 29 08:11:25 [conn5] replSet info voting yea for ip-10-2-11-210:31000 (0) m31002| Fri Mar 29 08:11:25 [conn2] replSet RECOVERING m31002| Fri Mar 29 08:11:25 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31000| Fri Mar 29 08:11:25 [rsMgr] replSet PRIMARY m31001| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state RECOVERING m31001| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31002| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31002| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state ARBITER m31000| Fri Mar 29 08:11:27 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state RECOVERING m31000| Fri Mar 29 08:11:27 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state ARBITER Fri Mar 29 08:11:32 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544692698), ok: 1.0 } Fri Mar 29 08:11:32 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544692699), ok: 1.0 } Fri Mar 29 08:11:33 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544693699), ok: 1.0 } Fri Mar 29 08:11:33 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "secOnlyNoSec", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544693700), ok: 1.0 } m31000| Fri Mar 29 08:11:34 [conn5] end connection 10.2.11.210:58290 (4 connections now open) m31000| Fri Mar 29 08:11:34 [initandlisten] connection accepted from 10.2.11.210:58294 #7 (5 connections now open) m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync pending m31002| Fri Mar 29 08:11:34 [rsSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:34 [initandlisten] connection accepted from 10.2.11.210:58295 #8 (6 connections now open) m31002| Fri Mar 29 08:11:34 [rsSync] build index local.me { _id: 1 } m31002| Fri Mar 29 08:11:34 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync drop all databases m31002| Fri Mar 29 08:11:34 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync clone all databases m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync data copy, starting syncup m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 1 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 2 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync building indexes m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 3 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync finishing up m31002| Fri Mar 29 08:11:34 [rsSync] replSet set minValid=51554c92:1 m31002| Fri Mar 29 08:11:34 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Fri Mar 29 08:11:34 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync done m31000| Fri Mar 29 08:11:34 [conn8] end connection 10.2.11.210:58295 (5 connections now open) Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "secOnlyNoSec", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544694700), ok: 1.0 } m31002| Fri Mar 29 08:11:35 [rsBackgroundSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:35 [initandlisten] connection accepted from 10.2.11.210:58296 #9 (6 connections now open) m31002| Fri Mar 29 08:11:35 [rsSyncNotifier] replset setting oplog notifier to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:35 [initandlisten] connection accepted from 10.2.11.210:58297 #10 (7 connections now open) Fri Mar 29 08:11:35 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "secOnlyNoSec", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544695701), ok: 1.0 } m31000| Fri Mar 29 08:11:36 [slaveTracking] build index local.slaves { _id: 1 } m31000| Fri Mar 29 08:11:36 [slaveTracking] build index done. scanned 0 total records. 0.001 secs m31002| Fri Mar 29 08:11:36 [rsSync] replSet SECONDARY Fri Mar 29 08:11:36 starting new replica set monitor for replica set secOnlyNoSec with seed of ip-10-2-11-210:31000,ip-10-2-11-210:31001,ip-10-2-11-210:31002 m31000| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:58298 #11 (8 connections now open) Fri Mar 29 08:11:36 successfully connected to seed ip-10-2-11-210:31000 for replica set secOnlyNoSec Fri Mar 29 08:11:36 changing hosts to { 0: "ip-10-2-11-210:31000", 1: "ip-10-2-11-210:31002" } from secOnlyNoSec/ Fri Mar 29 08:11:36 trying to add new host ip-10-2-11-210:31000 to replica set secOnlyNoSec m31000| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:58299 #12 (9 connections now open) Fri Mar 29 08:11:36 successfully connected to new host ip-10-2-11-210:31000 in replica set secOnlyNoSec Fri Mar 29 08:11:36 trying to add new host ip-10-2-11-210:31002 to replica set secOnlyNoSec m31002| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:47915 #4 (4 connections now open) Fri Mar 29 08:11:36 successfully connected to new host ip-10-2-11-210:31002 in replica set secOnlyNoSec Fri Mar 29 08:11:36 DBClientCursor::init call() failed m31001| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:59763 #6 (5 connections now open) m31000| Fri Mar 29 08:11:36 [conn11] end connection 10.2.11.210:58298 (8 connections now open) Fri Mar 29 08:11:36 successfully connected to seed ip-10-2-11-210:31001 for replica set secOnlyNoSec m31001| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:59764 #7 (6 connections now open) m31001| Fri Mar 29 08:11:36 [conn6] end connection 10.2.11.210:59763 (5 connections now open) m31000| Fri Mar 29 08:11:36 [initandlisten] connection accepted from 10.2.11.210:58303 #13 (9 connections now open) Fri Mar 29 08:11:36 Primary for replica set secOnlyNoSec changed to ip-10-2-11-210:31000 Fri Mar 29 08:11:36 DBClientCursor::init call() failed Fri Mar 29 08:11:36 replica set monitor for replica set secOnlyNoSec started, address is secOnlyNoSec/ip-10-2-11-210:31000,ip-10-2-11-210:31002 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number ReplSetTest stop *** Shutting down mongod in port 31002 *** m31000| Fri Mar 29 08:11:36 [conn7] end connection 10.2.11.210:58294 (8 connections now open) m31002| Fri Mar 29 08:11:36 got signal 15 (Terminated), will terminate after current cmd ends m31002| Fri Mar 29 08:11:36 [interruptThread] now exiting m31002| Fri Mar 29 08:11:36 dbexit: m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: going to close listening sockets... m31002| Fri Mar 29 08:11:36 [interruptThread] closing listening socket: 29 m31002| Fri Mar 29 08:11:36 [interruptThread] closing listening socket: 30 m31002| Fri Mar 29 08:11:36 [interruptThread] closing listening socket: 31 m31002| Fri Mar 29 08:11:36 [interruptThread] removing socket file: /tmp/mongodb-31002.sock m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: going to flush diaglog... m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: going to close sockets... m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: waiting for fs preallocator... m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: lock for final commit... m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: final commit... m31001| Fri Mar 29 08:11:36 [conn4] end connection 10.2.11.210:59753 (4 connections now open) m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: closing all files... m31002| Fri Mar 29 08:11:36 [interruptThread] closeAllFiles() finished m31002| Fri Mar 29 08:11:36 [interruptThread] journalCleanup... m31002| Fri Mar 29 08:11:36 [interruptThread] removeJournalFiles m31002| Fri Mar 29 08:11:36 [interruptThread] shutdown: removing fs lock... m31002| Fri Mar 29 08:11:36 dbexit: really exiting now Fri Mar 29 08:11:36 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:11:36 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks Fri Mar 29 08:11:36 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31000| Fri Mar 29 08:11:37 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31002: couldn't connect to server ip-10-2-11-210:31002 m31000| Fri Mar 29 08:11:37 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31002: couldn't connect to server ip-10-2-11-210:31002 m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet info ip-10-2-11-210:31002 is down (or slow to respond): socket exception [CONNECT_ERROR] for ip-10-2-11-210:31002 m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state DOWN Fri Mar 29 08:11:37 shell: stopped mongo program on port 31002 Fri Mar 29 08:11:37 DBClientCursor::init call() failed Fri Mar 29 08:11:37 query failed : admin.$cmd { ismaster: 1.0 } to: 127.0.0.1:31002 ReplSetTest Could not call ismaster on node 2 m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58307 #14 (9 connections now open) m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58308 #15 (10 connections now open) m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58309 #16 (11 connections now open) m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58310 #17 (12 connections now open) ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** m31001| Fri Mar 29 08:11:37 [conn5] end connection 10.2.11.210:59754 (3 connections now open) m31000| Fri Mar 29 08:11:37 got signal 15 (Terminated), will terminate after current cmd ends m31000| Fri Mar 29 08:11:37 [interruptThread] now exiting m31000| Fri Mar 29 08:11:37 dbexit: m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to close listening sockets... m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 22 m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 23 m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 25 m31000| Fri Mar 29 08:11:37 [interruptThread] removing socket file: /tmp/mongodb-31000.sock m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to flush diaglog... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to close sockets... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: waiting for fs preallocator... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: lock for final commit... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: final commit... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: closing all files... m31000| Fri Mar 29 08:11:37 [interruptThread] closeAllFiles() finished m31000| Fri Mar 29 08:11:37 [interruptThread] journalCleanup... m31000| Fri Mar 29 08:11:37 [interruptThread] removeJournalFiles m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: removing fs lock... m31000| Fri Mar 29 08:11:37 dbexit: really exiting now m31001| Fri Mar 29 08:11:38 [rsHealthPoll] DBClientCursor::init call() failed m31001| Fri Mar 29 08:11:38 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31002: couldn't connect to server ip-10-2-11-210:31002 m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet info ip-10-2-11-210:31000 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31000 ns: admin.$cmd query: { replSetHeartbeat: "secOnlyNoSec", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31001", $auth: {} } m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state DOWN m31001| Fri Mar 29 08:11:38 [rsHealthPoll] couldn't connect to ip-10-2-11-210:31002: couldn't connect to server ip-10-2-11-210:31002 m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet info ip-10-2-11-210:31002 is down (or slow to respond): socket exception [CONNECT_ERROR] for ip-10-2-11-210:31002 m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state DOWN Fri Mar 29 08:11:38 shell: stopped mongo program on port 31000 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number ReplSetTest stop *** Shutting down mongod in port 31001 *** m31001| Fri Mar 29 08:11:38 got signal 15 (Terminated), will terminate after current cmd ends m31001| Fri Mar 29 08:11:38 [interruptThread] now exiting m31001| Fri Mar 29 08:11:38 dbexit: m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: going to close listening sockets... m31001| Fri Mar 29 08:11:38 [interruptThread] closing listening socket: 26 m31001| Fri Mar 29 08:11:38 [interruptThread] closing listening socket: 27 m31001| Fri Mar 29 08:11:38 [interruptThread] closing listening socket: 28 m31001| Fri Mar 29 08:11:38 [interruptThread] removing socket file: /tmp/mongodb-31001.sock m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: going to flush diaglog... m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: going to close sockets... m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: waiting for fs preallocator... m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: lock for final commit... m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: final commit... m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: closing all files... m31001| Fri Mar 29 08:11:38 [interruptThread] closeAllFiles() finished m31001| Fri Mar 29 08:11:38 [interruptThread] journalCleanup... m31001| Fri Mar 29 08:11:38 [interruptThread] removeJournalFiles m31001| Fri Mar 29 08:11:38 [interruptThread] shutdown: removing fs lock... m31001| Fri Mar 29 08:11:38 dbexit: really exiting now Fri Mar 29 08:11:39 shell: stopped mongo program on port 31001 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number ReplSetTest stop *** Shutting down mongod in port 31002 *** Fri Mar 29 08:11:39 No db started on port: 31002 Fri Mar 29 08:11:39 shell: stopped mongo program on port 31002 ReplSetTest stopSet deleting all dbpaths ReplSetTest stopSet *** Shut down repl set - test worked **** ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priPrefNoPri", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "priPrefNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priPrefNoPri-0' Fri Mar 29 08:11:39 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31000 --noprealloc --smallfiles --rest --replSet priPrefNoPri --dbpath /data/db/priPrefNoPri-0 m31000| note: noprealloc may hurt performance in many applications m31000| Fri Mar 29 08:11:39 [initandlisten] MongoDB starting : pid=20793 port=31000 dbpath=/data/db/priPrefNoPri-0 64-bit host=ip-10-2-11-210 m31000| Fri Mar 29 08:11:39 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31000| Fri Mar 29 08:11:39 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31000| Fri Mar 29 08:11:39 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31000| Fri Mar 29 08:11:39 [initandlisten] options: { dbpath: "/data/db/priPrefNoPri-0", noprealloc: true, oplogSize: 1, port: 31000, replSet: "priPrefNoPri", rest: true, smallfiles: true } m31000| Fri Mar 29 08:11:39 [initandlisten] journal dir=/data/db/priPrefNoPri-0/journal m31000| Fri Mar 29 08:11:39 [initandlisten] recover : no journal files present, no recovery needed m31000| Fri Mar 29 08:11:39 [initandlisten] waiting for connections on port 31000 m31000| Fri Mar 29 08:11:39 [websvr] admin web console waiting for connections on port 32000 m31000| Fri Mar 29 08:11:39 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Fri Mar 29 08:11:39 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Fri Mar 29 08:11:40 [initandlisten] connection accepted from 127.0.0.1:40160 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priPrefNoPri", "dbpath" : "$set-$node", "arbiter" : true, "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "priPrefNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priPrefNoPri-1' Fri Mar 29 08:11:40 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31001 --noprealloc --smallfiles --rest --replSet priPrefNoPri --dbpath /data/db/priPrefNoPri-1 m31001| note: noprealloc may hurt performance in many applications m31001| Fri Mar 29 08:11:40 [initandlisten] MongoDB starting : pid=20841 port=31001 dbpath=/data/db/priPrefNoPri-1 64-bit host=ip-10-2-11-210 m31001| Fri Mar 29 08:11:40 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31001| Fri Mar 29 08:11:40 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31001| Fri Mar 29 08:11:40 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31001| Fri Mar 29 08:11:40 [initandlisten] options: { dbpath: "/data/db/priPrefNoPri-1", noprealloc: true, oplogSize: 1, port: 31001, replSet: "priPrefNoPri", rest: true, smallfiles: true } m31001| Fri Mar 29 08:11:40 [initandlisten] journal dir=/data/db/priPrefNoPri-1/journal m31001| Fri Mar 29 08:11:40 [initandlisten] recover : no journal files present, no recovery needed m31001| Fri Mar 29 08:11:40 [initandlisten] waiting for connections on port 31001 m31001| Fri Mar 29 08:11:40 [websvr] admin web console waiting for connections on port 32001 m31001| Fri Mar 29 08:11:40 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Fri Mar 29 08:11:40 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Fri Mar 29 08:11:40 [initandlisten] connection accepted from 127.0.0.1:37804 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 1, "keyFile" : undefined, "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "priPrefNoPri", "dbpath" : "$set-$node", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "priPrefNoPri" } } ReplSetTest Starting.... Resetting db path '/data/db/priPrefNoPri-2' Fri Mar 29 08:11:40 shell: started program /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/mongod --oplogSize 1 --port 31002 --noprealloc --smallfiles --rest --replSet priPrefNoPri --dbpath /data/db/priPrefNoPri-2 m31002| note: noprealloc may hurt performance in many applications m31002| Fri Mar 29 08:11:40 [initandlisten] MongoDB starting : pid=20889 port=31002 dbpath=/data/db/priPrefNoPri-2 64-bit host=ip-10-2-11-210 m31002| Fri Mar 29 08:11:40 [initandlisten] db version v2.2.4-rc1-pre-, pdfile version 4.5 m31002| Fri Mar 29 08:11:40 [initandlisten] git version: d1421e517c41af36cd756f591b913e3c2b3fed76 m31002| Fri Mar 29 08:11:40 [initandlisten] build info: Linux ip-10-2-11-210 3.0.13-0.27-ec2 #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 BOOST_LIB_VERSION=1_49 m31002| Fri Mar 29 08:11:40 [initandlisten] options: { dbpath: "/data/db/priPrefNoPri-2", noprealloc: true, oplogSize: 1, port: 31002, replSet: "priPrefNoPri", rest: true, smallfiles: true } m31002| Fri Mar 29 08:11:40 [initandlisten] journal dir=/data/db/priPrefNoPri-2/journal m31002| Fri Mar 29 08:11:40 [initandlisten] recover : no journal files present, no recovery needed m31002| Fri Mar 29 08:11:40 [initandlisten] waiting for connections on port 31002 m31002| Fri Mar 29 08:11:40 [websvr] admin web console waiting for connections on port 32002 m31002| Fri Mar 29 08:11:40 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Fri Mar 29 08:11:40 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Fri Mar 29 08:11:40 [initandlisten] connection accepted from 127.0.0.1:62743 #1 (1 connection now open) [ connection to ip-10-2-11-210:31000, connection to ip-10-2-11-210:31001, connection to ip-10-2-11-210:31002 ] { "replSetInitiate" : { "_id" : "priPrefNoPri", "members" : [ { "_id" : 0, "host" : "ip-10-2-11-210:31000" }, { "_id" : 1, "host" : "ip-10-2-11-210:31001", "arbiterOnly" : true }, { "_id" : 2, "host" : "ip-10-2-11-210:31002" } ] } } m31000| Fri Mar 29 08:11:40 [conn1] replSet replSetInitiate admin command received from client m31000| Fri Mar 29 08:11:40 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Fri Mar 29 08:11:40 [initandlisten] connection accepted from 10.2.11.210:59785 #2 (2 connections now open) m31002| Fri Mar 29 08:11:40 [initandlisten] connection accepted from 10.2.11.210:47939 #2 (2 connections now open) m31000| Fri Mar 29 08:11:40 [conn1] replSet replSetInitiate all members seem up m31000| Fri Mar 29 08:11:40 [conn1] ****** m31000| Fri Mar 29 08:11:40 [conn1] creating replication oplog of size: 1MB... m31000| Fri Mar 29 08:11:40 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-0/local.ns, filling with zeroes... m31000| Fri Mar 29 08:11:40 [FileAllocator] creating directory /data/db/priPrefNoPri-0/_tmp m31000| Fri Mar 29 08:11:40 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-0/local.ns, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:11:40 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-0/local.0, filling with zeroes... m31000| Fri Mar 29 08:11:40 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-0/local.0, size: 16MB, took 0.001 secs m31000| Fri Mar 29 08:11:40 [conn1] ****** m31000| Fri Mar 29 08:11:40 [conn1] replSet info saving a newer config version to local.system.replset m31000| Fri Mar 29 08:11:40 [conn1] replSet saveConfigLocally done m31000| Fri Mar 29 08:11:40 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } Fri Mar 29 08:11:46 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:11:46 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:15 [initandlisten] connection accepted from 10.2.11.210:58325 #2 (2 connections now open) Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544675868), ok: 1.0 } Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31001 m31001| Fri Mar 29 08:11:15 [initandlisten] connection accepted from 10.2.11.210:59788 #3 (3 connections now open) Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31001 ok Fri Mar 29 08:11:15 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544675869), ok: 1.0 } Fri Mar 29 08:11:16 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:11:16 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks Fri Mar 29 08:11:16 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:11:17 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:17 [initandlisten] connection accepted from 10.2.11.210:58327 #3 (3 connections now open) Fri Mar 29 08:11:17 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:11:17 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { ismaster: false, secondary: false, info: "Received replSetInitiate - should come online shortly.", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544677870), ok: 1.0 } m31000| Fri Mar 29 08:11:18 [rsStart] replSet I am ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:18 [rsStart] replSet STARTUP2 m31000| Fri Mar 29 08:11:18 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31000| Fri Mar 29 08:11:18 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31001| Fri Mar 29 08:11:18 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:58328 #4 (4 connections now open) m31001| Fri Mar 29 08:11:18 [rsStart] replSet I am ip-10-2-11-210:31001 m31001| Fri Mar 29 08:11:18 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Fri Mar 29 08:11:18 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-1/local.ns, filling with zeroes... m31001| Fri Mar 29 08:11:18 [FileAllocator] creating directory /data/db/priPrefNoPri-1/_tmp m31001| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-1/local.ns, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-1/local.0, filling with zeroes... m31001| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-1/local.0, size: 16MB, took 0.002 secs m31001| Fri Mar 29 08:11:18 [rsStart] replSet saveConfigLocally done m31001| Fri Mar 29 08:11:18 [rsStart] replSet STARTUP2 Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:58329 #5 (5 connections now open) Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31000 ok Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544678871), ok: 1.0 } Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] trying reconnect to ip-10-2-11-210:31002 m31002| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:47945 #3 (3 connections now open) Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] reconnect ip-10-2-11-210:31002 ok Fri Mar 29 08:11:18 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31002 isn't a part of set: secOnlyNoSec ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1364544678872), ok: 1.0 } m31002| Fri Mar 29 08:11:18 [rsStart] trying to contact ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:18 [initandlisten] connection accepted from 10.2.11.210:58331 #6 (6 connections now open) m31002| Fri Mar 29 08:11:18 [rsStart] replSet I am ip-10-2-11-210:31002 m31002| Fri Mar 29 08:11:18 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Fri Mar 29 08:11:18 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-2/local.ns, filling with zeroes... m31002| Fri Mar 29 08:11:18 [FileAllocator] creating directory /data/db/priPrefNoPri-2/_tmp m31002| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-2/local.ns, size: 16MB, took 0.002 secs m31002| Fri Mar 29 08:11:18 [FileAllocator] allocating new datafile /data/db/priPrefNoPri-2/local.0, filling with zeroes... m31002| Fri Mar 29 08:11:18 [FileAllocator] done allocating datafile /data/db/priPrefNoPri-2/local.0, size: 16MB, took 0.001 secs m31002| Fri Mar 29 08:11:18 [rsStart] replSet saveConfigLocally done m31002| Fri Mar 29 08:11:18 [rsStart] replSet STARTUP2 m31002| Fri Mar 29 08:11:18 [rsSync] ****** m31002| Fri Mar 29 08:11:18 [rsSync] creating replication oplog of size: 1MB... m31002| Fri Mar 29 08:11:18 [rsSync] ****** m31002| Fri Mar 29 08:11:18 [rsSync] replSet initial sync pending m31002| Fri Mar 29 08:11:18 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Fri Mar 29 08:11:19 [rsSync] replSet SECONDARY Fri Mar 29 08:11:19 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544679872), ok: 1.0 } Fri Mar 29 08:11:19 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31002 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31002", maxBsonObjectSize: 16777216, localTime: new Date(1364544679873), ok: 1.0 } m31000| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 m31000| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:11:20 [rsMgr] not electing self, ip-10-2-11-210:31002 would veto m31000| Fri Mar 29 08:11:20 [rsMgr] not electing self, ip-10-2-11-210:31002 would veto m31001| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31001| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31002| Fri Mar 29 08:11:20 [initandlisten] connection accepted from 10.2.11.210:47947 #4 (4 connections now open) m31001| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31001| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state STARTUP2 Fri Mar 29 08:11:20 [ReplicaSetMonitorWatcher] warning: No primary detected for set secOnlyNoSec Fri Mar 29 08:11:20 [ReplicaSetMonitorWatcher] All nodes for set secOnlyNoSec are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is up m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state SECONDARY m31001| Fri Mar 29 08:11:20 [initandlisten] connection accepted from 10.2.11.210:59795 #4 (4 connections now open) m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is up m31002| Fri Mar 29 08:11:20 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state STARTUP2 m31000| Fri Mar 29 08:11:26 [rsMgr] replSet info electSelf 0 m31002| Fri Mar 29 08:11:26 [conn2] replSet RECOVERING m31002| Fri Mar 29 08:11:26 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31001| Fri Mar 29 08:11:26 [conn2] replSet RECOVERING m31001| Fri Mar 29 08:11:26 [conn2] replSet info voting yea for ip-10-2-11-210:31000 (0) m31000| Fri Mar 29 08:11:26 [rsMgr] replSet PRIMARY m31001| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state RECOVERING m31001| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31002| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state PRIMARY m31002| Fri Mar 29 08:11:26 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state ARBITER m31000| Fri Mar 29 08:11:28 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state ARBITER m31000| Fri Mar 29 08:11:28 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state RECOVERING Fri Mar 29 08:11:30 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544690873), ok: 1.0 } Fri Mar 29 08:11:30 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544690874), ok: 1.0 } Fri Mar 29 08:11:31 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: basic ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544691874), ok: 1.0 } Fri Mar 29 08:11:31 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31001 isn't a part of set: basic ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", arbiterOnly: true, me: "ip-10-2-11-210:31001", maxBsonObjectSize: 16777216, localTime: new Date(1364544691875), ok: 1.0 } m31001| Fri Mar 29 08:11:32 [conn2] end connection 10.2.11.210:59785 (3 connections now open) m31001| Fri Mar 29 08:11:32 [initandlisten] connection accepted from 10.2.11.210:59796 #5 (4 connections now open) Fri Mar 29 08:11:32 [ReplicaSetMonitorWatcher] warning: No primary detected for set basic Fri Mar 29 08:11:32 [ReplicaSetMonitorWatcher] All nodes for set basic are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks Fri Mar 29 08:11:32 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544692875), ok: 1.0 } Fri Mar 29 08:11:33 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: priOnlyNoPri ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544693876), ok: 1.0 } m31000| Fri Mar 29 08:11:34 [conn4] end connection 10.2.11.210:58328 (5 connections now open) m31000| Fri Mar 29 08:11:34 [initandlisten] connection accepted from 10.2.11.210:58335 #7 (7 connections now open) Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] warning: No primary detected for set priOnlyNoPri Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] All nodes for set priOnlyNoPri are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544694876), ok: 1.0 } Fri Mar 29 08:11:34 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31002 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31002", maxBsonObjectSize: 16777216, localTime: new Date(1364544694876), ok: 1.0 } m31000| Fri Mar 29 08:11:34 [initandlisten] connection accepted from 10.2.11.210:58336 #8 (7 connections now open) m31000| Fri Mar 29 08:11:34 [conn6] end connection 10.2.11.210:58331 (5 connections now open) m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync pending m31002| Fri Mar 29 08:11:34 [rsSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:34 [initandlisten] connection accepted from 10.2.11.210:58337 #9 (7 connections now open) m31002| Fri Mar 29 08:11:34 [rsSync] build index local.me { _id: 1 } m31002| Fri Mar 29 08:11:34 [rsSync] build index done. scanned 0 total records. 0.001 secs m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync drop all databases m31002| Fri Mar 29 08:11:34 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync clone all databases m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync data copy, starting syncup m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 1 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 2 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync building indexes m31002| Fri Mar 29 08:11:34 [rsSync] oplog sync 3 of 3 m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync finishing up m31002| Fri Mar 29 08:11:34 [rsSync] replSet set minValid=51554cbc:1 m31002| Fri Mar 29 08:11:34 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Fri Mar 29 08:11:34 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Fri Mar 29 08:11:34 [rsSync] replSet initial sync done m31000| Fri Mar 29 08:11:34 [conn9] end connection 10.2.11.210:58337 (6 connections now open) Fri Mar 29 08:11:35 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31000 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31000", "ip-10-2-11-210:31002" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31000", maxBsonObjectSize: 16777216, localTime: new Date(1364544695877), ok: 1.0 } Fri Mar 29 08:11:35 [ReplicaSetMonitorWatcher] warning: node: ip-10-2-11-210:31002 isn't a part of set: secOnlyNoSec ismaster: { setName: "priPrefNoPri", ismaster: false, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31000", me: "ip-10-2-11-210:31002", maxBsonObjectSize: 16777216, localTime: new Date(1364544695877), ok: 1.0 } m31002| Fri Mar 29 08:11:35 [rsBackgroundSync] replSet syncing to: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:35 [initandlisten] connection accepted from 10.2.11.210:58338 #10 (7 connections now open) m31002| Fri Mar 29 08:11:35 [rsSyncNotifier] replset setting oplog notifier to ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:35 [initandlisten] connection accepted from 10.2.11.210:58339 #11 (8 connections now open) Fri Mar 29 08:11:36 [ReplicaSetMonitorWatcher] warning: No primary detected for set secOnlyNoSec Fri Mar 29 08:11:36 [ReplicaSetMonitorWatcher] All nodes for set secOnlyNoSec are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks m31000| Fri Mar 29 08:11:36 [slaveTracking] build index local.slaves { _id: 1 } m31000| Fri Mar 29 08:11:36 [slaveTracking] build index done. scanned 0 total records. 0 secs m31002| Fri Mar 29 08:11:36 [rsSync] replSet SECONDARY m31000| Fri Mar 29 08:11:37 [conn1] replSet replSetReconfig config object parses ok, 3 members specified m31000| Fri Mar 29 08:11:37 [conn1] replSet replSetReconfig [2] m31000| Fri Mar 29 08:11:37 [conn1] replSet info saving a newer config version to local.system.replset Fri Mar 29 08:11:37 DBClientCursor::init call() failed Fri Mar 29 08:11:37 query failed : admin.$cmd { replSetReconfig: { _id: "priPrefNoPri", version: 2, members: [ { _id: 0, host: "ip-10-2-11-210:31000", priority: 99.0 }, { _id: 1, host: "ip-10-2-11-210:31001", arbiterOnly: true }, { _id: 2, host: "ip-10-2-11-210:31002", priority: 0.0 } ] } } to: 127.0.0.1:31000 Exception from reconfig: Error: error doing query: failed Fri Mar 29 08:11:37 starting new replica set monitor for replica set priPrefNoPri with seed of ip-10-2-11-210:31000,ip-10-2-11-210:31001,ip-10-2-11-210:31002 m31002| Fri Mar 29 08:11:37 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: ip-10-2-11-210:31000 m31002| Fri Mar 29 08:11:37 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-10-2-11-210:31000 m31000| Fri Mar 29 08:11:37 [conn1] clock skew detected prev: 1364544700 now: 1364544697 m31000| Fri Mar 29 08:11:37 [conn1] replSet saveConfigLocally done m31000| Fri Mar 29 08:11:37 [conn1] replSet relinquishing primary state m31000| Fri Mar 29 08:11:37 [conn1] replSet SECONDARY m31000| Fri Mar 29 08:11:37 [conn1] replSet closing client sockets after relinquishing primary m31000| Fri Mar 29 08:11:37 [conn1] replSet PRIMARY m31000| Fri Mar 29 08:11:37 [conn1] replSet replSetReconfig new config saved locally m31000| Fri Mar 29 08:11:37 [conn1] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:40160] m31002| Fri Mar 29 08:11:37 [conn2] end connection 10.2.11.210:47939 (3 connections now open) m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet info ip-10-2-11-210:31001 is down (or slow to respond): socket exception [SEND_ERROR] for 10.2.11.210:31001 m31001| Fri Mar 29 08:11:37 [conn5] end connection 10.2.11.210:59796 (3 connections now open) m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet member ip-10-2-11-210:31001 is now in state DOWN m31000| Fri Mar 29 08:11:37 [conn10] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.2.11.210:58338] m31000| Fri Mar 29 08:11:37 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.2.11.210:58339] m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58340 #12 (7 connections now open) Fri Mar 29 08:11:37 successfully connected to seed ip-10-2-11-210:31000 for replica set priPrefNoPri m31002| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:47956 #5 (4 connections now open) Fri Mar 29 08:11:37 changing hosts to { 0: "ip-10-2-11-210:31000", 1: "ip-10-2-11-210:31002" } from priPrefNoPri/ Fri Mar 29 08:11:37 trying to add new host ip-10-2-11-210:31000 to replica set priPrefNoPri m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is up m31000| Fri Mar 29 08:11:37 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state SECONDARY m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58342 #13 (7 connections now open) Fri Mar 29 08:11:37 successfully connected to new host ip-10-2-11-210:31000 in replica set priPrefNoPri Fri Mar 29 08:11:37 trying to add new host ip-10-2-11-210:31002 to replica set priPrefNoPri m31002| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:47958 #6 (5 connections now open) Fri Mar 29 08:11:37 successfully connected to new host ip-10-2-11-210:31002 in replica set priPrefNoPri Fri Mar 29 08:11:37 DBClientCursor::init call() failed m31000| Fri Mar 29 08:11:37 [conn12] end connection 10.2.11.210:58340 (6 connections now open) m31001| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:59806 #6 (4 connections now open) Fri Mar 29 08:11:37 successfully connected to seed ip-10-2-11-210:31001 for replica set priPrefNoPri Fri Mar 29 08:11:37 DBClientCursor::init call() failed m31001| Fri Mar 29 08:11:37 [conn6] end connection 10.2.11.210:59806 (3 connections now open) m31000| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:58345 #14 (7 connections now open) Fri Mar 29 08:11:37 Primary for replica set priPrefNoPri changed to ip-10-2-11-210:31000 m31002| Fri Mar 29 08:11:37 [initandlisten] connection accepted from 10.2.11.210:47961 #7 (6 connections now open) Fri Mar 29 08:11:37 replica set monitor for replica set priPrefNoPri started, address is priPrefNoPri/ip-10-2-11-210:31000,ip-10-2-11-210:31002 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number ReplSetTest stop *** Shutting down mongod in port 31000 *** m31000| Fri Mar 29 08:11:37 got signal 15 (Terminated), will terminate after current cmd ends m31000| Fri Mar 29 08:11:37 [interruptThread] now exiting m31000| Fri Mar 29 08:11:37 dbexit: m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to close listening sockets... m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 34 m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 35 m31000| Fri Mar 29 08:11:37 [interruptThread] closing listening socket: 36 m31000| Fri Mar 29 08:11:37 [interruptThread] removing socket file: /tmp/mongodb-31000.sock m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to flush diaglog... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: going to close sockets... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: waiting for fs preallocator... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: lock for final commit... m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: final commit... m31002| Fri Mar 29 08:11:37 [conn5] end connection 10.2.11.210:47956 (5 connections now open) m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: closing all files... m31000| Fri Mar 29 08:11:37 [interruptThread] closeAllFiles() finished m31000| Fri Mar 29 08:11:37 [interruptThread] journalCleanup... m31000| Fri Mar 29 08:11:37 [interruptThread] removeJournalFiles m31000| Fri Mar 29 08:11:37 [interruptThread] shutdown: removing fs lock... m31000| Fri Mar 29 08:11:37 dbexit: really exiting now Fri Mar 29 08:11:38 shell: stopped mongo program on port 31000 Fri Mar 29 08:11:38 DBClientCursor::init call() failed Fri Mar 29 08:11:38 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:11:38 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 m31001| Fri Mar 29 08:11:38 [rsHealthPoll] DBClientCursor::init call() failed m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet info ip-10-2-11-210:31000 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31000 ns: admin.$cmd query: { replSetHeartbeat: "priPrefNoPri", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31001", $auth: {} } m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state DOWN m31001| Fri Mar 29 08:11:38 [rsHealthPoll] replSet member ip-10-2-11-210:31002 is now in state SECONDARY m31002| Fri Mar 29 08:11:38 [rsHealthPoll] DBClientCursor::init call() failed m31002| Fri Mar 29 08:11:38 [rsHealthPoll] replSet info ip-10-2-11-210:31000 is down (or slow to respond): DBClientBase::findN: transport error: ip-10-2-11-210:31000 ns: admin.$cmd query: { replSetHeartbeat: "priPrefNoPri", v: 1, pv: 1, checkEmpty: false, from: "ip-10-2-11-210:31002", $auth: {} } m31002| Fri Mar 29 08:11:38 [rsHealthPoll] replSet member ip-10-2-11-210:31000 is now in state DOWN m31002| Fri Mar 29 08:11:38 [rsMgr] not electing self, ip-10-2-11-210:31001 would veto Fri Mar 29 08:11:39 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [FAILED_STATE] for ip-10-2-11-210:31000 Fri Mar 29 08:11:39 ReplicaSetMonitor::_checkConnection: ip-10-2-11-210:31002 { setName: "priPrefNoPri", ismaster: false, secondary: true, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], me: "ip-10-2-11-210:31002", maxBsonObjectSize: 16777216, localTime: new Date(1364544699091), ok: 1.0 } Fri Mar 29 08:12:11 warning: No primary detected for set priPrefNoPri Fri Mar 29 08:12:11 trying reconnect to ip-10-2-11-210:31000 Fri Mar 29 08:12:11 reconnect ip-10-2-11-210:31000 failed couldn't connect to server ip-10-2-11-210:31000 m31002| Fri Mar 29 08:12:12 [rsMgr] replSet info electSelf 2 m31001| Fri Mar 29 08:12:12 [conn4] replSet info voting yea for ip-10-2-11-210:31002 (2) m31002| Fri Mar 29 08:12:12 [rsMgr] replSet PRIMARY Fri Mar 29 08:12:12 ReplicaSetMonitor::_checkConnection: caught exception ip-10-2-11-210:31000 socket exception [FAILED_STATE] for ip-10-2-11-210:31000 Fri Mar 29 08:12:12 ReplicaSetMonitor::_checkConnection: ip-10-2-11-210:31002 { setName: "priPrefNoPri", ismaster: true, secondary: false, hosts: [ "ip-10-2-11-210:31002", "ip-10-2-11-210:31000" ], arbiters: [ "ip-10-2-11-210:31001" ], primary: "ip-10-2-11-210:31002", me: "ip-10-2-11-210:31002", maxBsonObjectSize: 16777216, localTime: new Date(1364544732932), ok: 1.0 } Fri Mar 29 08:12:12 Primary for replica set priPrefNoPri changed to ip-10-2-11-210:31002 m31002| Fri Mar 29 08:12:12 [initandlisten] connection accepted from 10.2.11.210:47967 #8 (6 connections now open) assert: did not throw exception: undefined Error("Printing Stack Trace")@:0 ()@src/mongo/shell/utils.js:37 ("did not throw exception: undefined")@src/mongo/shell/utils.js:58 ((function () {coll.find().readPref("primary").explain();}))@src/mongo/shell/utils.js:204 noPriSecOkTest()@/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js:179 @/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js:218 Fri Mar 29 08:12:12 uncaught exception: did not throw exception: undefined failed to load: /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js 149161.700964ms Fri Mar 29 08:12:14 got signal 15 (Terminated), will terminate after current cmd ends Fri Mar 29 08:12:14 [interruptThread] now exiting Fri Mar 29 08:12:14 dbexit: Fri Mar 29 08:12:14 [interruptThread] shutdown: going to close listening sockets... Fri Mar 29 08:12:14 [interruptThread] closing listening socket: 6 Fri Mar 29 08:12:14 [interruptThread] closing listening socket: 7 Fri Mar 29 08:12:14 [interruptThread] closing listening socket: 8 Fri Mar 29 08:12:14 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Fri Mar 29 08:12:14 [interruptThread] shutdown: going to flush diaglog... Fri Mar 29 08:12:14 [interruptThread] shutdown: going to close sockets... Fri Mar 29 08:12:14 [interruptThread] shutdown: waiting for fs preallocator... Fri Mar 29 08:12:14 [interruptThread] shutdown: lock for final commit... Fri Mar 29 08:12:14 [interruptThread] shutdown: final commit... Fri Mar 29 08:12:14 [interruptThread] shutdown: closing all files... Fri Mar 29 08:12:14 [interruptThread] closeAllFiles() finished Fri Mar 29 08:12:14 [interruptThread] journalCleanup... Fri Mar 29 08:12:14 [interruptThread] removeJournalFiles Fri Mar 29 08:12:14 [interruptThread] shutdown: removing fs lock... Fri Mar 29 08:12:14 dbexit: really exiting now test /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js exited with status 253 78 tests succeeded 40 tests didn't get run The following tests failed (with exit code): /data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/jstests/sharding/read_pref_rs_client.js 253 Traceback (most recent call last): File "/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/buildscripts/smoke.py", line 861, in main() File "/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/buildscripts/smoke.py", line 857, in main report() File "/data/buildslaves/Linux_64bit_SSL_SUSE_11_V2.2/mongo/buildscripts/smoke.py", line 546, in report raise Exception("Test failures") Exception: Test failures scons: *** [smokeSharding] Error 1 scons: building terminated because of errors.