Tue Oct 30 03:53:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:53:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f495a9ac1f416901503
Tue Oct 30 03:53:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:53:53 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f505a9ac1f416901504
Tue Oct 30 03:53:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f585a9ac1f416901505
Tue Oct 30 03:54:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f5e5a9ac1f416901506
Tue Oct 30 03:54:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:12 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f645a9ac1f416901507
Tue Oct 30 03:54:12 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:19 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f6a5a9ac1f416901508
Tue Oct 30 03:54:19 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f715a9ac1f416901509
Tue Oct 30 03:54:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f785a9ac1f41690150a
Tue Oct 30 03:54:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f7f5a9ac1f41690150b
Tue Oct 30 03:54:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f855a9ac1f41690150c
Tue Oct 30 03:54:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:54:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f8c5a9ac1f41690150d
Tue Oct 30 03:54:53 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f935a9ac1f41690150e
Tue Oct 30 03:55:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4f9a5a9ac1f41690150f
Tue Oct 30 03:55:07 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fa15a9ac1f416901510
Tue Oct 30 03:55:14 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:20 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fa85a9ac1f416901511
Tue Oct 30 03:55:20 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fae5a9ac1f416901512
Tue Oct 30 03:55:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fb55a9ac1f416901513
Tue Oct 30 03:55:34 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:41 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fbc5a9ac1f416901514
Tue Oct 30 03:55:41 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:55:53 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fc95a9ac1f416901515
Tue Oct 30 03:55:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fd65a9ac1f416901516
Tue Oct 30 03:56:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:12 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fdc5a9ac1f416901517
Tue Oct 30 03:56:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:25 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fe95a9ac1f416901518
Tue Oct 30 03:56:25 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4fef5a9ac1f416901519
Tue Oct 30 03:56:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4ff75a9ac1f41690151a
Tue Oct 30 03:56:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f4ffd5a9ac1f41690151b
Tue Oct 30 03:56:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 03:56:54 [ReplicaSetMonitorWatcher] Socket recv() timeout  10.252.81.110:27018
Tue Oct 30 03:56:54 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [3] server [10.252.81.110:27018] 
Tue Oct 30 03:56:54 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 03:56:54 [ReplicaSetMonitorWatcher] trying reconnect to r1.n1.mongo.domain.com:27018
Tue Oct 30 03:56:54 [ReplicaSetMonitorWatcher] reconnect r1.n1.mongo.domain.com:27018 ok
Tue Oct 30 03:57:24 [CheckConfigServers] Socket recv() timeout  10.252.81.110:27019
Tue Oct 30 03:57:24 [CheckConfigServers] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] 
Tue Oct 30 03:57:24 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 03:57:24 [CheckConfigServers] warning:  couldn't check on config server:c1.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c1.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 03:57:34 [LockPinger] Socket recv() timeout  10.252.81.110:27019
Tue Oct 30 03:57:34 [LockPinger] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] 
Tue Oct 30 03:57:34 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 03:57:54 [CheckConfigServers] Socket recv() timeout  10.248.68.14:27019
Tue Oct 30 03:57:54 [CheckConfigServers] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] 
Tue Oct 30 03:57:54 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 03:57:54 [CheckConfigServers] warning:  couldn't check on config server:c2.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c2.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 03:58:04 [LockPinger] Socket recv() timeout  10.248.68.14:27019
Tue Oct 30 03:58:04 [LockPinger] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] 
Tue Oct 30 03:58:04 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 03:58:24 [CheckConfigServers] Socket recv() timeout  10.252.23.239:27019
Tue Oct 30 03:58:24 [CheckConfigServers] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] 
Tue Oct 30 03:58:24 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 03:58:24 [CheckConfigServers] warning:  couldn't check on config server:c3.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c3.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 03:58:24 [CheckConfigServers] ERROR: config servers not in sync! no config servers reachable
Tue Oct 30 03:58:34 [LockPinger] Socket recv() timeout  10.252.23.239:27019
Tue Oct 30 03:58:34 [LockPinger] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] 
Tue Oct 30 03:58:34 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 03:58:34 [LockPinger] scoped connection to c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 not being returned to the pool
Tue Oct 30 03:58:34 [LockPinger] warning: distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 10276 DBClientBase::findN: transport error: c1.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c1.mongo.domain.com:27019:{}10276 DBClientBase::findN: transport error: c2.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c2.mongo.domain.com:27019:{}10276 DBClientBase::findN: transport error: c3.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c3.mongo.domain.com:27019:{}
Tue Oct 30 03:59:04 [LockPinger] SyncClusterConnection connecting to [c1.mongo.domain.com:27019]
Tue Oct 30 03:59:04 [LockPinger] SyncClusterConnection connecting to [c2.mongo.domain.com:27019]
Tue Oct 30 03:59:04 [LockPinger] SyncClusterConnection connecting to [c3.mongo.domain.com:27019]
Tue Oct 30 03:59:04 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 03:59:04 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:03:35 [LockPinger] Socket recv() timeout  10.252.81.110:27019
Tue Oct 30 04:03:35 [LockPinger] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] 
Tue Oct 30 04:03:35 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 04:03:54 [CheckConfigServers] Socket recv() timeout  10.252.81.110:27019
Tue Oct 30 04:03:54 [CheckConfigServers] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] 
Tue Oct 30 04:03:54 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 04:03:54 [CheckConfigServers] warning:  couldn't check on config server:c1.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c1.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 04:04:05 [LockPinger] Socket recv() timeout  10.248.68.14:27019
Tue Oct 30 04:04:05 [LockPinger] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] 
Tue Oct 30 04:04:05 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 04:04:24 [CheckConfigServers] Socket recv() timeout  10.248.68.14:27019
Tue Oct 30 04:04:24 [CheckConfigServers] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] 
Tue Oct 30 04:04:24 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 04:04:24 [CheckConfigServers] warning:  couldn't check on config server:c2.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c2.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 04:04:35 [LockPinger] Socket recv() timeout  10.252.23.239:27019
Tue Oct 30 04:04:35 [LockPinger] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] 
Tue Oct 30 04:04:35 [LockPinger] DBClientCursor::init call() failed
Tue Oct 30 04:04:35 [LockPinger] scoped connection to c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 not being returned to the pool
Tue Oct 30 04:04:35 [LockPinger] warning: distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 10276 DBClientBase::findN: transport error: c1.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c1.mongo.domain.com:27019:{}10276 DBClientBase::findN: transport error: c2.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c2.mongo.domain.com:27019:{}10276 DBClientBase::findN: transport error: c3.mongo.domain.com:27019 ns: admin.$cmd query: { fsync: 1 } c3.mongo.domain.com:27019:{}
Tue Oct 30 04:04:54 [CheckConfigServers] Socket recv() timeout  10.252.23.239:27019
Tue Oct 30 04:04:54 [CheckConfigServers] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] 
Tue Oct 30 04:04:54 [CheckConfigServers] DBClientCursor::init call() failed
Tue Oct 30 04:04:54 [CheckConfigServers] warning:  couldn't check on config server:c3.mongo.domain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c3.mongo.domain.com:27019 ns: admin.$cmd query: { getlasterror: 1 }
Tue Oct 30 04:04:54 [CheckConfigServers] ERROR: config servers not in sync! no config servers reachable
Tue Oct 30 04:05:05 [LockPinger] SyncClusterConnection connecting to [c1.mongo.domain.com:27019]
Tue Oct 30 04:05:05 [LockPinger] SyncClusterConnection connecting to [c2.mongo.domain.com:27019]
Tue Oct 30 04:05:05 [LockPinger] SyncClusterConnection connecting to [c3.mongo.domain.com:27019]
Tue Oct 30 04:06:06 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 04:06:06 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:11:08 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 04:11:07 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:12:22 [Balancer] Socket recv() errno:110 Connection timed out 10.252.81.110:27019
Tue Oct 30 04:12:22 [Balancer] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [1] server [10.252.81.110:27019] 
Tue Oct 30 04:12:22 [Balancer] DBClientCursor::init call() failed
Tue Oct 30 04:12:22 [Balancer] query failed to: c1.mongo.domain.com:27019 no data
Tue Oct 30 04:12:25 [ReplicaSetMonitorWatcher] Socket recv() errno:110 Connection timed out 10.252.81.110:27018
Tue Oct 30 04:12:25 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [1] server [10.252.81.110:27018] 
Tue Oct 30 04:12:25 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:12:29 [conn1] Socket recv() errno:110 Connection timed out 10.248.108.136:27018
Tue Oct 30 04:12:29 [conn1] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [1] server [10.248.108.136:27018] 
Tue Oct 30 04:12:29 [conn1] DBClientCursor::init lazy say() failed
Tue Oct 30 04:12:29 [conn1] DBClientCursor::init message from say() was empty
Tue Oct 30 04:12:29 [conn1] got not master for: r2.n2.mongo.domain.com:27018
Tue Oct 30 04:12:29 [conn1] warning: db exception when finishing on replSet2:replSet2/r2.n1.mongo.domain.com:27018,r2.n2.mongo.domain.com:27018, current connection state is { state: { conn: "replSet2/r2.n1.mongo.domain.com:27018,r2.n2.mongo.domain.com:27018", vinfo: "replSet2:replSet2/r2.n1.mongo.domain.com:27018,r2.n2.mongo.domain.com:27018", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server
Tue Oct 30 04:12:30 [ReplicaSetMonitorWatcher] Socket recv() timeout  10.252.81.110:27018
Tue Oct 30 04:12:30 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [3] server [10.252.81.110:27018] 
Tue Oct 30 04:12:30 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:12:35 [ReplicaSetMonitorWatcher] Socket recv() timeout  10.248.68.14:27018
Tue Oct 30 04:12:35 [ReplicaSetMonitorWatcher] SocketException: remote: 10.248.68.14:27018 error: 9001 socket exception [3] server [10.248.68.14:27018] 
Tue Oct 30 04:12:35 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:12:40 [ReplicaSetMonitorWatcher] Socket recv() timeout  10.252.23.239:27018
Tue Oct 30 04:12:40 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [3] server [10.252.23.239:27018] 
Tue Oct 30 04:12:40 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:12:45 [ReplicaSetMonitorWatcher] Socket recv() timeout  10.248.108.136:27018
Tue Oct 30 04:12:45 [ReplicaSetMonitorWatcher] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [3] server [10.248.108.136:27018] 
Tue Oct 30 04:12:45 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:12:46 [ReplicaSetMonitorWatcher] trying reconnect to r2.n1.mongo.domain.com:27018
Tue Oct 30 04:12:46 [ReplicaSetMonitorWatcher] reconnect r2.n1.mongo.domain.com:27018 ok
Tue Oct 30 04:12:46 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: r2.n1.mongo.domain.com:27018 { setName: "replSet2", ismaster: false, secondary: true, hosts: [ "r2.n1.mongo.domain.com:27018", "r2.n2.mongo.domain.com:27018" ], arbiters: [ "r2.a1.mongo.domain.com:30000" ], primary: "r2.n2.mongo.domain.com:27018", me: "r2.n1.mongo.domain.com:27018", maxBsonObjectSize: 16777216, localTime: new Date(1351570368505), ok: 1.0 }
Tue Oct 30 04:12:53 [WriteBackListener-r1.n1.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.252.81.110:27018
Tue Oct 30 04:12:53 [WriteBackListener-r1.n1.mongo.domain.com:27018] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [1] server [10.252.81.110:27018] 
Tue Oct 30 04:12:53 [WriteBackListener-r1.n1.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:12:53 [WriteBackListener-r1.n1.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n1.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.248.68.14:27018
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] SocketException: remote: 10.248.68.14:27018 error: 9001 socket exception [1] server [10.248.68.14:27018] 
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n2.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c1.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.248.108.136:27018
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [1] server [10.248.108.136:27018] 
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n2.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c1.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.252.23.239:27018
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [1] server [10.252.23.239:27018] 
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n1.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] SyncClusterConnection connecting to [c1.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] SyncClusterConnection connecting to [c2.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c2.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c2.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n1.mongo.domain.com:27018] SyncClusterConnection connecting to [c3.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r1.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c3.mongo.domain.com:27019]
Tue Oct 30 04:12:53 [WriteBackListener-r2.n2.mongo.domain.com:27018] SyncClusterConnection connecting to [c3.mongo.domain.com:27019]
Tue Oct 30 04:16:10 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 04:16:10 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:21:13 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 04:21:13 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:26:17 [LockPinger] cluster c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 pinged successfully at Tue Oct 30 04:26:17 2012 by distributed lock pinger 'c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019/ip-10-252-160-159:27017:1351560050:1804289383', sleeping for 30000ms
Tue Oct 30 04:28:05 [Balancer] Socket recv() errno:110 Connection timed out 10.248.68.14:27019
Tue Oct 30 04:28:05 [Balancer] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [1] server [10.248.68.14:27019] 
Tue Oct 30 04:28:05 [Balancer] DBClientCursor::init call() failed
Tue Oct 30 04:28:05 [Balancer] query failed to: c2.mongo.domain.com:27019 no data
Tue Oct 30 04:28:05 [Balancer] Socket say send() errno:110 Connection timed out 10.252.23.239:27019
Tue Oct 30 04:28:05 [Balancer] query failed to: c3.mongo.domain.com:27019 exception: socket exception [SEND_ERROR] for 10.252.23.239:27019
Tue Oct 30 04:28:05 [Balancer] scoped connection to c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 not being returned to the pool
Tue Oct 30 04:28:05 [Balancer] scoped connection to c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 not being returned to the pool
Tue Oct 30 04:28:05 [Balancer] caught exception while doing balance: all servers down!
Tue Oct 30 04:28:11 [Balancer] trying reconnect to r1.n1.mongo.domain.com:27018
Tue Oct 30 04:28:11 [Balancer] reconnect r1.n1.mongo.domain.com:27018 ok
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] Socket recv() errno:110 Connection timed out 10.252.23.239:27018
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [1] server [10.252.23.239:27018] 
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception r2.n1.mongo.domain.com:27018 DBClientBase::findN: transport error: r2.n1.mongo.domain.com:27018 ns: admin.$cmd query: { replSetGetStatus: 1 }
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] trying reconnect to r2.n2.mongo.domain.com:27018
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] reconnect r2.n2.mongo.domain.com:27018 ok
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] Primary for replica set replSet2 changed to r2.n2.mongo.domain.com:27018
Tue Oct 30 04:28:17 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: r2.n2.mongo.domain.com:27018 { setName: "replSet2", ismaster: true, secondary: false, hosts: [ "r2.n2.mongo.domain.com:27018", "r2.n1.mongo.domain.com:27018" ], arbiters: [ "r2.a1.mongo.domain.com:30000" ], primary: "r2.n2.mongo.domain.com:27018", me: "r2.n2.mongo.domain.com:27018", maxBsonObjectSize: 16777216, localTime: new Date(1351571298453), ok: 1.0 }
Tue Oct 30 04:28:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57615a9ac1f41690151c
Tue Oct 30 04:28:18 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57685a9ac1f41690151d
Tue Oct 30 04:28:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:25 [WriteBackListener-r2.n2.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.248.108.136:27018
Tue Oct 30 04:28:25 [WriteBackListener-r2.n2.mongo.domain.com:27018] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [1] server [10.248.108.136:27018] 
Tue Oct 30 04:28:25 [WriteBackListener-r2.n2.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:28:25 [WriteBackListener-r2.n2.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n2.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:28:25 [WriteBackListener-r1.n2.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.248.68.14:27018
Tue Oct 30 04:28:25 [WriteBackListener-r1.n2.mongo.domain.com:27018] SocketException: remote: 10.248.68.14:27018 error: 9001 socket exception [1] server [10.248.68.14:27018] 
Tue Oct 30 04:28:25 [WriteBackListener-r1.n2.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:28:25 [WriteBackListener-r1.n2.mongo.domain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n2.mongo.domain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('508f2b715a9ac1f41690105e') }
Tue Oct 30 04:28:27 [ReplicaSetMonitorWatcher] trying reconnect to r1.n2.mongo.domain.com:27018
Tue Oct 30 04:28:27 [ReplicaSetMonitorWatcher] reconnect r1.n2.mongo.domain.com:27018 ok
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] Socket recv() errno:110 Connection timed out 10.252.81.110:27019
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [1] server [10.252.81.110:27019] 
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] DBClientCursor::init call() failed
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] query failed to: c1.mongo.domain.com:27019 no data
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] Socket say send() errno:110 Connection timed out 10.248.68.14:27019
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] query failed to: c2.mongo.domain.com:27019 exception: socket exception [SEND_ERROR] for 10.248.68.14:27019
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] Socket say send() errno:110 Connection timed out 10.252.23.239:27019
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] query failed to: c3.mongo.domain.com:27019 exception: socket exception [SEND_ERROR] for 10.252.23.239:27019
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] scoped connection to c1.mongo.domain.com:27019,c2.mongo.domain.com:27019,c3.mongo.domain.com:27019 not being returned to the pool
Tue Oct 30 04:28:28 [WriteBackListener-r1.n1.mongo.domain.com:27018] ERROR: backgroundjob WriteBackListener-r1.n1.mongo.domain.com:27018error: all servers down!
Tue Oct 30 04:28:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f576e5a9ac1f41690151e
Tue Oct 30 04:28:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57745a9ac1f41690151f
Tue Oct 30 04:28:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f577b5a9ac1f416901520
Tue Oct 30 04:28:44 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:50 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57825a9ac1f416901521
Tue Oct 30 04:28:50 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:28:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57885a9ac1f416901522
Tue Oct 30 04:28:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f578f5a9ac1f416901523
Tue Oct 30 04:29:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57965a9ac1f416901524
Tue Oct 30 04:29:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f579c5a9ac1f416901525
Tue Oct 30 04:29:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:23 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57a35a9ac1f416901526
Tue Oct 30 04:29:23 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:29 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57a95a9ac1f416901527
Tue Oct 30 04:29:29 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:35 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57af5a9ac1f416901528
Tue Oct 30 04:29:36 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:42 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57b65a9ac1f416901529
Tue Oct 30 04:29:42 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:48 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57bc5a9ac1f41690152a
Tue Oct 30 04:29:48 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:29:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57c25a9ac1f41690152b
Tue Oct 30 04:29:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57c85a9ac1f41690152c
Tue Oct 30 04:30:00 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:07 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57ce5a9ac1f41690152d
Tue Oct 30 04:30:07 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57d55a9ac1f41690152e
Tue Oct 30 04:30:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:19 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57db5a9ac1f41690152f
Tue Oct 30 04:30:19 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57e15a9ac1f416901530
Tue Oct 30 04:30:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57e85a9ac1f416901531
Tue Oct 30 04:30:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57ee5a9ac1f416901532
Tue Oct 30 04:30:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57f55a9ac1f416901533
Tue Oct 30 04:30:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f57fb5a9ac1f416901534
Tue Oct 30 04:30:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:30:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f58025a9ac1f416901535
Tue Oct 30 04:30:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:31:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f58085a9ac1f416901536
Tue Oct 30 04:31:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. 
Tue Oct 30 04:31:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f580e5a9ac1f416901537