***** SERVER RESTARTED ***** Tue Jul 31 08:56:11 mongos db version v2.0.6, pdfile version 4.5 starting (--help for usage) Tue Jul 31 08:56:11 git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc Tue Jul 31 08:56:11 build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41 Tue Jul 31 08:56:11 config string : 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 Tue Jul 31 08:56:11 creating new connection to:10.136.20.57:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 connected connection! Tue Jul 31 08:56:11 creating new connection to:10.136.20.58:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 connected connection! Tue Jul 31 08:56:11 creating new connection to:10.136.20.59:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 connected connection! Tue Jul 31 08:56:11 SyncClusterConnection connecting to [10.136.20.57:20000] Tue Jul 31 08:56:11 BackgroundJob starting: CheckConfigServers Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 SyncClusterConnection connecting to [10.136.20.58:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 SyncClusterConnection connecting to [10.136.20.59:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 MaxChunkSize: 128 Tue Jul 31 08:56:11 BackgroundJob starting: Balancer Tue Jul 31 08:56:11 [Balancer] about to contact config servers and shards Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.57:20000] Tue Jul 31 08:56:11 BackgroundJob starting: cursorTimeout Tue Jul 31 08:56:11 [websvr] fd limit hard:204800 soft:204800 max conn: 163840 Tue Jul 31 08:56:11 [websvr] admin web console waiting for connections on port 31000 Tue Jul 31 08:56:11 [mongosMain] fd limit hard:204800 soft:204800 max conn: 163840 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [mongosMain] waiting for connections on port 30000 Tue Jul 31 08:56:11 BackgroundJob starting: PeriodicTask::Runner Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.58:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.59:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] starting new replica set monitor for replica set shard1 with seed of 10.136.16.67:27032,10.136.16.68:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to seed 10.136.16.67:27032 for replica set shard1 Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] changing hosts to { 0: "10.136.16.67:27032", 1: "10.136.16.73:27032", 2: "10.136.16.68:27032" } from shard1/ Tue Jul 31 08:56:11 [Balancer] trying to add new host 10.136.16.67:27032 to replica set shard1 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to new host 10.136.16.67:27032 in replica set shard1 Tue Jul 31 08:56:11 [Balancer] trying to add new host 10.136.16.68:27032 to replica set shard1 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to new host 10.136.16.68:27032 in replica set shard1 Tue Jul 31 08:56:11 [Balancer] trying to add new host 10.136.16.73:27032 to replica set shard1 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to new host 10.136.16.73:27032 in replica set shard1 Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.16.67:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.16.68:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.16.73:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] replica set monitor for replica set shard1 started, address is shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:11 [Balancer] starting new replica set monitor for replica set shard2 with seed of 10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ReplicaSetMonitorWatcher Tue Jul 31 08:56:11 [ReplicaSetMonitorWatcher] starting Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to seed 10.136.16.69:27032 for replica set shard2 Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] changing hosts to { 0: "10.136.16.69:27032", 1: "10.136.16.70:27032" } from shard2/ Tue Jul 31 08:56:11 [Balancer] trying to add new host 10.136.16.69:27032 to replica set shard2 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to new host 10.136.16.69:27032 in replica set shard2 Tue Jul 31 08:56:11 [Balancer] trying to add new host 10.136.16.70:27032 to replica set shard2 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] successfully connected to new host 10.136.16.70:27032 in replica set shard2 Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.16.69:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.16.70:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] replica set monitor for replica set shard2 started, address is shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] config servers and shards contacted successfully Tue Jul 31 08:56:11 [Balancer] balancer id: mongos-2059:30000 started at Jul 31 08:56:11 Tue Jul 31 08:56:11 [Balancer] created new distributed lock for balancer on 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.57:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.58:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.59:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.57:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.58:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] SyncClusterConnection connecting to [10.136.20.59:20000] Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.20.57:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.57:20000 found: -2 Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.20.58:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.58:20000 found: -2 Tue Jul 31 08:56:11 [Balancer] creating new connection to:10.136.20.59:20000 Tue Jul 31 08:56:11 BackgroundJob starting: ConnectBG Tue Jul 31 08:56:11 [Balancer] connected connection! Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.59:20000 found: -1 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.57:20000 found: -2 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.58:20000 found: -1 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.59:20000 found: 0 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.57:20000 found: -3 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.58:20000 found: -2 Tue Jul 31 08:56:11 [Balancer] skew from remote server 10.136.20.59:20000 found: 0 Tue Jul 31 08:56:11 [Balancer] total clock skew of 2ms for servers 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 is in 30000ms bounds. Tue Jul 31 08:56:11 [LockPinger] creating distributed lock ping thread for 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 and process mongos-2059:30000:1343696171:1804289383 (sleeping for 30000ms) Tue Jul 31 08:56:11 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:56:11 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:56:11 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d2b09d663852fb7b6ed" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d2a5dcba5206e56b68c" } } Tue Jul 31 08:56:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d2b09d663852fb7b6ed Tue Jul 31 08:56:11 [Balancer] *** start balancing round Tue Jul 31 08:56:11 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:56:11 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:56:11 [Balancer] collection : d_name.t_name Tue Jul 31 08:56:11 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:56:11 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:56:11 [Balancer] no need to move any chunk Tue Jul 31 08:56:11 [Balancer] *** end of balancing round Tue Jul 31 08:56:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:21 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:56:21 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d3509d663852fb7b6ee" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d345dcba5206e56b68d" } } Tue Jul 31 08:56:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d3509d663852fb7b6ee Tue Jul 31 08:56:21 [Balancer] *** start balancing round Tue Jul 31 08:56:21 [Balancer] collection : d_name.t_name Tue Jul 31 08:56:21 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:56:21 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:56:21 [Balancer] no need to move any chunk Tue Jul 31 08:56:21 [Balancer] *** end of balancing round Tue Jul 31 08:56:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:31 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:56:31 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d3f09d663852fb7b6ef" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d3e5dcba5206e56b68e" } } Tue Jul 31 08:56:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d3f09d663852fb7b6ef Tue Jul 31 08:56:31 [Balancer] *** start balancing round Tue Jul 31 08:56:31 [Balancer] collection : d_name.t_name Tue Jul 31 08:56:31 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:56:31 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:56:31 [Balancer] no need to move any chunk Tue Jul 31 08:56:31 [Balancer] *** end of balancing round Tue Jul 31 08:56:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:41 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:56:41 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:56:41 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:56:41 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:56:41 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d4909d663852fb7b6f0" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d485dcba5206e56b68f" } } Tue Jul 31 08:56:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d4909d663852fb7b6f0 Tue Jul 31 08:56:41 [Balancer] *** start balancing round Tue Jul 31 08:56:41 [Balancer] collection : d_name.t_name Tue Jul 31 08:56:41 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:56:41 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:56:41 [Balancer] no need to move any chunk Tue Jul 31 08:56:41 [Balancer] *** end of balancing round Tue Jul 31 08:56:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:56:51 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:56:51 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d5309d663852fb7b6f1" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d525dcba5206e56b690" } } Tue Jul 31 08:56:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d5309d663852fb7b6f1 Tue Jul 31 08:56:51 [Balancer] *** start balancing round Tue Jul 31 08:56:51 [Balancer] collection : d_name.t_name Tue Jul 31 08:56:51 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:56:51 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:56:51 [Balancer] no need to move any chunk Tue Jul 31 08:56:51 [Balancer] *** end of balancing round Tue Jul 31 08:56:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:01 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:01 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d5d09d663852fb7b6f2" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d5c5dcba5206e56b691" } } Tue Jul 31 08:57:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d5d09d663852fb7b6f2 Tue Jul 31 08:57:01 [Balancer] *** start balancing round Tue Jul 31 08:57:01 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:01 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:01 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:01 [Balancer] no need to move any chunk Tue Jul 31 08:57:01 [Balancer] *** end of balancing round Tue Jul 31 08:57:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:57:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:57:11 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:11 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:57:11 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:57:11 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:11 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d6709d663852fb7b6f3" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d665dcba5206e56b692" } } Tue Jul 31 08:57:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d6709d663852fb7b6f3 Tue Jul 31 08:57:11 [Balancer] *** start balancing round Tue Jul 31 08:57:11 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:11 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:11 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:11 [Balancer] no need to move any chunk Tue Jul 31 08:57:11 [Balancer] *** end of balancing round Tue Jul 31 08:57:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:21 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:21 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d7109d663852fb7b6f4" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d705dcba5206e56b693" } } Tue Jul 31 08:57:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d7109d663852fb7b6f4 Tue Jul 31 08:57:21 [Balancer] *** start balancing round Tue Jul 31 08:57:21 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:21 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:21 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:21 [Balancer] no need to move any chunk Tue Jul 31 08:57:21 [Balancer] *** end of balancing round Tue Jul 31 08:57:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:31 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:31 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d7b09d663852fb7b6f5" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d7a5dcba5206e56b694" } } Tue Jul 31 08:57:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d7b09d663852fb7b6f5 Tue Jul 31 08:57:31 [Balancer] *** start balancing round Tue Jul 31 08:57:31 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:31 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:31 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:31 [Balancer] no need to move any chunk Tue Jul 31 08:57:31 [Balancer] *** end of balancing round Tue Jul 31 08:57:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:41 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:57:41 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:57:41 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:41 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:41 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d8509d663852fb7b6f6" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d845dcba5206e56b695" } } Tue Jul 31 08:57:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d8509d663852fb7b6f6 Tue Jul 31 08:57:41 [Balancer] *** start balancing round Tue Jul 31 08:57:41 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:41 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:41 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:41 [Balancer] no need to move any chunk Tue Jul 31 08:57:41 [Balancer] *** end of balancing round Tue Jul 31 08:57:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:57:51 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:57:51 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d8f09d663852fb7b6f7" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d8e5dcba5206e56b696" } } Tue Jul 31 08:57:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d8f09d663852fb7b6f7 Tue Jul 31 08:57:51 [Balancer] *** start balancing round Tue Jul 31 08:57:51 [Balancer] collection : d_name.t_name Tue Jul 31 08:57:51 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:57:51 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:57:51 [Balancer] no need to move any chunk Tue Jul 31 08:57:51 [Balancer] *** end of balancing round Tue Jul 31 08:57:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:57:56 [mongosMain] connection accepted from 127.0.0.1:9927 #1 Tue Jul 31 08:57:56 [conn1] DBConfig unserialize: admin { _id: "admin", partitioned: false, primary: "config" } Tue Jul 31 08:57:56 [conn1] Request::process ns: admin.$cmd msg id:0 attempt: 0 Tue Jul 31 08:57:56 [conn1] single query: admin.$cmd { whatsmyuri: 1 } ntoreturn: 1 options : 0 Tue Jul 31 08:57:56 [conn1] Request::process ns: admin.$cmd msg id:1 attempt: 0 Tue Jul 31 08:57:56 [conn1] single query: admin.$cmd { replSetGetStatus: 1, forShell: 1 } ntoreturn: 1 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: config.version msg id:2 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.version {} ntoreturn: -1 options : 0 Tue Jul 31 08:57:58 BackgroundJob starting: ConnectBG Tue Jul 31 08:57:58 [conn1] creating WriteBackListener for: 10.136.16.67:27032 serverID: 50172d2b09d663852fb7b6ec Tue Jul 31 08:57:58 [conn1] creating WriteBackListener for: 10.136.16.68:27032 serverID: 50172d2b09d663852fb7b6ec Tue Jul 31 08:57:58 [conn1] creating WriteBackListener for: 10.136.16.73:27032 serverID: 50172d2b09d663852fb7b6ec Tue Jul 31 08:57:58 BackgroundJob starting: WriteBackListener-10.136.16.67:27032 Tue Jul 31 08:57:58 BackgroundJob starting: WriteBackListener-10.136.16.68:27032 Tue Jul 31 08:57:58 [conn1] initializing shard connection to 10.136.16.67:27032 Tue Jul 31 08:57:58 [conn1] initial sharding settings : { setShardVersion: "", init: true, configdb: "10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000", serverID: ObjectId('50172d2b09d663852fb7b6ec'), authoritative: true } Tue Jul 31 08:57:58 BackgroundJob starting: WriteBackListener-10.136.16.73:27032 Tue Jul 31 08:57:58 [conn1] initial sharding result : { initialized: true, ok: 1.0 } Tue Jul 31 08:57:58 BackgroundJob starting: ConnectBG Tue Jul 31 08:57:58 [conn1] creating WriteBackListener for: 10.136.16.69:27032 serverID: 50172d2b09d663852fb7b6ec Tue Jul 31 08:57:58 [conn1] creating WriteBackListener for: 10.136.16.70:27032 serverID: 50172d2b09d663852fb7b6ec Tue Jul 31 08:57:58 BackgroundJob starting: WriteBackListener-10.136.16.69:27032 Tue Jul 31 08:57:58 [conn1] initializing shard connection to 10.136.16.69:27032 Tue Jul 31 08:57:58 [conn1] initial sharding settings : { setShardVersion: "", init: true, configdb: "10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000", serverID: ObjectId('50172d2b09d663852fb7b6ec'), authoritative: true } Tue Jul 31 08:57:58 BackgroundJob starting: WriteBackListener-10.136.16.70:27032 Tue Jul 31 08:57:58 [conn1] initial sharding result : { initialized: true, ok: 1.0 } Tue Jul 31 08:57:58 [conn1] SyncClusterConnection connecting to [10.136.20.57:20000] Tue Jul 31 08:57:58 BackgroundJob starting: ConnectBG Tue Jul 31 08:57:58 [conn1] SyncClusterConnection connecting to [10.136.20.58:20000] Tue Jul 31 08:57:58 BackgroundJob starting: ConnectBG Tue Jul 31 08:57:58 [conn1] SyncClusterConnection connecting to [10.136.20.59:20000] Tue Jul 31 08:57:58 BackgroundJob starting: ConnectBG Tue Jul 31 08:57:58 [conn1] Request::process ns: config.version msg id:3 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.version {} ntoreturn: -1 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: config.shards msg id:4 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.shards { query: {}, orderby: { _id: 1.0 } } ntoreturn: 0 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: config.databases msg id:5 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.databases { query: {}, orderby: { name: 1.0 } } ntoreturn: 0 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: config.collections msg id:6 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.collections { query: { _id: /^d_name\./ }, orderby: { _id: 1.0 } } ntoreturn: 0 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: config.$cmd msg id:7 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: config.$cmd { group: { cond: { ns: "d_name.t_name" }, key: { shard: 1.0 }, initial: { nChunks: 0.0 }, ns: "chunks", $reduce: function (doc, out) { out.nChunks++; } } } ntoreturn: -1 options : 0 Tue Jul 31 08:57:58 [conn1] DBConfig unserialize: test { _id: "test", partitioned: false, primary: "shard1" } Tue Jul 31 08:57:58 [conn1] Request::process ns: test.$cmd msg id:8 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: test.$cmd { getlasterror: 1.0, w: 1.0 } ntoreturn: -1 options : 0 Tue Jul 31 08:57:58 [conn1] Request::process ns: admin.$cmd msg id:9 attempt: 0 Tue Jul 31 08:57:58 [conn1] single query: admin.$cmd { replSetGetStatus: 1, forShell: 1 } ntoreturn: 1 options : 0 Tue Jul 31 08:58:00 [conn1] Socket recv() conn closed? 127.0.0.1:9927 Tue Jul 31 08:58:00 [conn1] SocketException: remote: 127.0.0.1:9927 error: 9001 socket exception [0] server [127.0.0.1:9927] Tue Jul 31 08:58:00 [conn1] end connection 127.0.0.1:9927 Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] creating new connection to:10.136.16.67:27032 Tue Jul 31 08:58:01 BackgroundJob starting: ConnectBG Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] connected connection! Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] creating new connection to:10.136.16.68:27032 Tue Jul 31 08:58:01 BackgroundJob starting: ConnectBG Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] connected connection! Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] creating new connection to:10.136.16.73:27032 Tue Jul 31 08:58:01 BackgroundJob starting: ConnectBG Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] connected connection! Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] creating new connection to:10.136.16.69:27032 Tue Jul 31 08:58:01 BackgroundJob starting: ConnectBG Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] connected connection! Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] creating new connection to:10.136.16.70:27032 Tue Jul 31 08:58:01 BackgroundJob starting: ConnectBG Tue Jul 31 08:58:01 [ReplicaSetMonitorWatcher] connected connection! Tue Jul 31 08:58:01 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:01 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172d9909d663852fb7b6f8" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172d985dcba5206e56b697" } } Tue Jul 31 08:58:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172d9909d663852fb7b6f8 Tue Jul 31 08:58:01 [Balancer] *** start balancing round Tue Jul 31 08:58:01 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:01 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:01 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:01 [Balancer] no need to move any chunk Tue Jul 31 08:58:01 [Balancer] *** end of balancing round Tue Jul 31 08:58:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:58:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:58:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:58:11 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:58:11 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:58:11 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:11 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:11 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172da309d663852fb7b6f9" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172da25dcba5206e56b698" } } Tue Jul 31 08:58:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172da309d663852fb7b6f9 Tue Jul 31 08:58:11 [Balancer] *** start balancing round Tue Jul 31 08:58:11 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:11 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:11 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:11 [Balancer] no need to move any chunk Tue Jul 31 08:58:11 [Balancer] *** end of balancing round Tue Jul 31 08:58:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:21 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:21 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172dad09d663852fb7b6fa" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172dac5dcba5206e56b699" } } Tue Jul 31 08:58:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172dad09d663852fb7b6fa Tue Jul 31 08:58:21 [Balancer] *** start balancing round Tue Jul 31 08:58:21 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:21 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:21 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:21 [Balancer] no need to move any chunk Tue Jul 31 08:58:21 [Balancer] *** end of balancing round Tue Jul 31 08:58:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:31 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:31 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172db709d663852fb7b6fb" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172db65dcba5206e56b69a" } } Tue Jul 31 08:58:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172db709d663852fb7b6fb Tue Jul 31 08:58:31 [Balancer] *** start balancing round Tue Jul 31 08:58:31 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:31 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:31 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:31 [Balancer] no need to move any chunk Tue Jul 31 08:58:31 [Balancer] *** end of balancing round Tue Jul 31 08:58:31 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:58:41 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:58:41 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:58:41 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:41 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:41 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172dc109d663852fb7b6fc" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172dc05dcba5206e56b69b" } } Tue Jul 31 08:58:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172dc109d663852fb7b6fc Tue Jul 31 08:58:41 [Balancer] *** start balancing round Tue Jul 31 08:58:41 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:41 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:41 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:41 [Balancer] no need to move any chunk Tue Jul 31 08:58:41 [Balancer] *** end of balancing round Tue Jul 31 08:58:41 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:58:51 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:58:51 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172dcb09d663852fb7b6fd" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172dca5dcba5206e56b69c" } } Tue Jul 31 08:58:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172dcb09d663852fb7b6fd Tue Jul 31 08:58:51 [Balancer] *** start balancing round Tue Jul 31 08:58:51 [Balancer] collection : d_name.t_name Tue Jul 31 08:58:51 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:58:51 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:58:51 [Balancer] no need to move any chunk Tue Jul 31 08:58:51 [Balancer] *** end of balancing round Tue Jul 31 08:58:51 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:01 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:59:01 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172dd509d663852fb7b6fe" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172dd45dcba5206e56b69d" } } Tue Jul 31 08:59:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172dd509d663852fb7b6fe Tue Jul 31 08:59:01 [Balancer] *** start balancing round Tue Jul 31 08:59:01 [Balancer] collection : d_name.t_name Tue Jul 31 08:59:01 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:59:01 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:59:01 [Balancer] no need to move any chunk Tue Jul 31 08:59:01 [Balancer] *** end of balancing round Tue Jul 31 08:59:01 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:59:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:59:11 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms Tue Jul 31 08:59:11 [LockPinger] distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383' about to ping. Tue Jul 31 08:59:11 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Tue Jul 31 08:59:11 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2059:30000:1343696171:1804289383', sleeping for 30000ms Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:11 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:59:11 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172ddf09d663852fb7b6ff" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172dde5dcba5206e56b69e" } } Tue Jul 31 08:59:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172ddf09d663852fb7b6ff Tue Jul 31 08:59:11 [Balancer] *** start balancing round Tue Jul 31 08:59:11 [Balancer] collection : d_name.t_name Tue Jul 31 08:59:11 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:59:11 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:59:11 [Balancer] no need to move any chunk Tue Jul 31 08:59:11 [Balancer] *** end of balancing round Tue Jul 31 08:59:11 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked. Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] checking replica set: shard1 Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] _check : shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032 Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.67:27032 { setName: "shard1", ismaster: true, secondary: false, hosts: [ "10.136.16.67:27032", "10.136.16.73:27032", "10.136.16.68:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.67:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.68:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.68:27032", "10.136.16.73:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.68:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.73:27032 { setName: "shard1", ismaster: false, secondary: true, hosts: [ "10.136.16.73:27032", "10.136.16.68:27032", "10.136.16.67:27032" ], arbiters: [ "10.136.16.68:37032" ], primary: "10.136.16.67:27032", me: "10.136.16.73:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] checking replica set: shard2 Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] _check : shard2/10.136.16.69:27032,10.136.16.70:27032 Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.69:27032 { setName: "shard2", ismaster: true, secondary: false, hosts: [ "10.136.16.69:27032", "10.136.16.70:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.69:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: 10.136.16.70:27032 { setName: "shard2", ismaster: false, secondary: true, hosts: [ "10.136.16.70:27032", "10.136.16.69:27032" ], arbiters: [ "10.136.16.70:37032" ], primary: "10.136.16.69:27032", me: "10.136.16.70:27032", maxBsonObjectSize: 16777216, ok: 1.0 } Tue Jul 31 08:59:21 [Balancer] about to acquire distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383: { "state" : 1, "who" : "mongos-2059:30000:1343696171:1804289383:Balancer:846930886", "process" : "mongos-2059:30000:1343696171:1804289383", "when" : { "$date" : "Tue Jul 31 08:59:21 2012" }, "why" : "doing balance round", "ts" : { "$oid" : "50172de909d663852fb7b700" } } { "_id" : "balancer", "state" : 0, "ts" : { "$oid" : "50172de85dcba5206e56b69f" } } Tue Jul 31 08:59:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' acquired, ts : 50172de909d663852fb7b700 Tue Jul 31 08:59:21 [Balancer] *** start balancing round Tue Jul 31 08:59:21 [Balancer] collection : d_name.t_name Tue Jul 31 08:59:21 [Balancer] donor : 245 chunks on shard1 Tue Jul 31 08:59:21 [Balancer] receiver : 244 chunks on shard2 Tue Jul 31 08:59:21 [Balancer] no need to move any chunk Tue Jul 31 08:59:21 [Balancer] *** end of balancing round Tue Jul 31 08:59:21 [Balancer] distributed lock 'balancer/mongos-2059:30000:1343696171:1804289383' unlocked.