Tue Oct 30 00:46:21 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f236344e086d5bb077149 Tue Oct 30 00:46:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f236944e086d5bb07714a Tue Oct 30 00:46:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f236f44e086d5bb07714b Tue Oct 30 00:46:40 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f237644e086d5bb07714c Tue Oct 30 00:46:47 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:53 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f237d44e086d5bb07714d Tue Oct 30 00:46:53 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:46:59 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f238344e086d5bb07714e Tue Oct 30 00:46:59 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:05 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f238944e086d5bb07714f Tue Oct 30 00:47:06 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f239044e086d5bb077150 Tue Oct 30 00:47:13 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:19 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f239744e086d5bb077151 Tue Oct 30 00:47:20 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f239e44e086d5bb077152 Tue Oct 30 00:47:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23a444e086d5bb077153 Tue Oct 30 00:47:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23aa44e086d5bb077154 Tue Oct 30 00:47:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23b144e086d5bb077155 Tue Oct 30 00:47:45 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23b744e086d5bb077156 Tue Oct 30 00:47:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:47:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23be44e086d5bb077157 Tue Oct 30 00:47:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:05 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23c444e086d5bb077158 Tue Oct 30 00:48:05 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:08 [conn82] end connection 127.0.0.1:53158 (2 connections now open) Tue Oct 30 00:48:08 [conn83] end connection 127.0.0.1:37056 (1 connection now open) Tue Oct 30 00:48:12 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23cb44e086d5bb077159 Tue Oct 30 00:48:12 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:13 [mongosMain] connection accepted from 127.0.0.1:39145 #85 (2 connections now open) Tue Oct 30 00:48:13 [conn85] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "18596745b3b13aa8", user: "produser", key: "2741cb41180747cee2415ef0e89fd701" } Tue Oct 30 00:48:13 [conn85] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "ba51f6d5937448ef", key: "f56fdc361adba40787e2a719014dd005" } Tue Oct 30 00:48:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23d844e086d5bb07715a Tue Oct 30 00:48:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:30 [Balancer] could not acquire lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' (another update won) Tue Oct 30 00:48:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' was not acquired. Tue Oct 30 00:48:36 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23e444e086d5bb07715c Tue Oct 30 00:48:36 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23ea44e086d5bb07715d Tue Oct 30 00:48:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:49 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23f144e086d5bb07715e Tue Oct 30 00:48:49 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:48:55 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23f744e086d5bb07715f Tue Oct 30 00:48:55 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:02 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f23fd44e086d5bb077160 Tue Oct 30 00:49:02 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:08 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f240444e086d5bb077161 Tue Oct 30 00:49:08 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:14 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f240a44e086d5bb077162 Tue Oct 30 00:49:14 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:20 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f241044e086d5bb077163 Tue Oct 30 00:49:21 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f241744e086d5bb077164 Tue Oct 30 00:49:27 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f241d44e086d5bb077165 Tue Oct 30 00:49:33 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:36 [LockPinger] cluster c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 pinged successfully at Tue Oct 30 00:49:36 2012 by distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383', sleeping for 30000ms Tue Oct 30 00:49:40 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f242344e086d5bb077166 Tue Oct 30 00:49:40 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f242a44e086d5bb077167 Tue Oct 30 00:49:46 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f243044e086d5bb077168 Tue Oct 30 00:49:52 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:49:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f243644e086d5bb077169 Tue Oct 30 00:49:58 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:11 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f244244e086d5bb07716a Tue Oct 30 00:50:11 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f244944e086d5bb07716b Tue Oct 30 00:50:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f244f44e086d5bb07716c Tue Oct 30 00:50:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f245644e086d5bb07716d Tue Oct 30 00:50:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f245c44e086d5bb07716e Tue Oct 30 00:50:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f246344e086d5bb07716f Tue Oct 30 00:50:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:50:49 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' acquired, ts : 508f246944e086d5bb077170 Tue Oct 30 00:50:50 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1350954554:1804289383' unlocked. Tue Oct 30 00:51:04 [ReplicaSetMonitorWatcher] Socket recv() timeout 10.252.81.110:27018 Tue Oct 30 00:51:04 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [3] server [10.252.81.110:27018] Tue Oct 30 00:51:04 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 00:51:04 [ReplicaSetMonitorWatcher] trying reconnect to r1.n1.mongo.mydomain.com:27018 Tue Oct 30 00:51:04 [ReplicaSetMonitorWatcher] reconnect r1.n1.mongo.mydomain.com:27018 ok Tue Oct 30 00:51:36 [LockPinger] Socket recv() timeout 10.252.81.110:27019 Tue Oct 30 00:51:36 [LockPinger] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] Tue Oct 30 00:51:36 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:51:55 [CheckConfigServers] Socket recv() timeout 10.252.81.110:27019 Tue Oct 30 00:51:55 [CheckConfigServers] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] Tue Oct 30 00:51:55 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:51:55 [CheckConfigServers] warning: couldn't check on config server:c1.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c1.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:52:06 [LockPinger] Socket recv() timeout 10.248.68.14:27019 Tue Oct 30 00:52:06 [LockPinger] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] Tue Oct 30 00:52:06 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:52:25 [CheckConfigServers] Socket recv() timeout 10.248.68.14:27019 Tue Oct 30 00:52:25 [CheckConfigServers] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] Tue Oct 30 00:52:25 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:52:25 [CheckConfigServers] warning: couldn't check on config server:c2.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c2.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:52:36 [LockPinger] Socket recv() timeout 10.252.23.239:27019 Tue Oct 30 00:52:36 [LockPinger] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] Tue Oct 30 00:52:36 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:52:36 [LockPinger] scoped connection to c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 not being returned to the pool Tue Oct 30 00:52:36 [LockPinger] warning: distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 10276 DBClientBase::findN: transport error: c1.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c1.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c2.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c2.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c3.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c3.mongo.mydomain.com:27019:{} Tue Oct 30 00:52:55 [CheckConfigServers] Socket recv() timeout 10.252.23.239:27019 Tue Oct 30 00:52:55 [CheckConfigServers] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] Tue Oct 30 00:52:55 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:52:55 [CheckConfigServers] warning: couldn't check on config server:c3.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c3.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:52:55 [CheckConfigServers] ERROR: config servers not in sync! no config servers reachable Tue Oct 30 00:53:36 [LockPinger] Socket recv() timeout 10.252.81.110:27019 Tue Oct 30 00:53:36 [LockPinger] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] Tue Oct 30 00:53:36 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:53:53 [mongosMain] connection accepted from 127.0.0.1:39223 #86 (3 connections now open) Tue Oct 30 00:54:06 [LockPinger] Socket recv() timeout 10.248.68.14:27019 Tue Oct 30 00:54:06 [LockPinger] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] Tue Oct 30 00:54:06 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:54:23 [conn85] authenticate db: myapp2-prod-giveable_access_tokens { authenticate: 1, nonce: "8bd216470e27273f", user: "produser", key: "50f9c601d128dc0f5b119a308b98fe27" } Tue Oct 30 00:54:36 [LockPinger] Socket recv() timeout 10.252.23.239:27019 Tue Oct 30 00:54:36 [LockPinger] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] Tue Oct 30 00:54:36 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:54:36 [LockPinger] scoped connection to c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 not being returned to the pool Tue Oct 30 00:54:36 [LockPinger] warning: distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 10276 DBClientBase::findN: transport error: c1.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c1.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c2.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c2.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c3.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c3.mongo.mydomain.com:27019:{} Tue Oct 30 00:54:53 [conn85] Socket recv() timeout 10.252.23.239:27018 Tue Oct 30 00:54:53 [conn85] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [3] server [10.252.23.239:27018] Tue Oct 30 00:54:53 [conn85] DBClientCursor::init call() failed Tue Oct 30 00:54:53 [conn85] authenticate db: myapp2-prod-giveable_access_tokens { authenticate: 1, user: "produser", nonce: "260e17b1533f30d2", key: "5da4374b81fe1459483d7bf1e4bd710c" } Tue Oct 30 00:55:00 [mongosMain] connection accepted from 127.0.0.1:39259 #87 (4 connections now open) Tue Oct 30 00:55:00 [conn87] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "3624729fdad88acf", user: "produser", key: "b8ef23bfae8e5dd413fb90c7ecea5e7e" } Tue Oct 30 00:55:00 [conn87] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "b2f7bdc687707eba", key: "08cd8bfdb51bbf4b638f463358a78745" } Tue Oct 30 00:55:06 [LockPinger] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 00:55:06 [LockPinger] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 00:55:06 [LockPinger] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 00:55:34 [mongosMain] connection accepted from 127.0.0.1:39277 #88 (5 connections now open) Tue Oct 30 00:55:34 [conn88] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "d757afe0b5b96d0", user: "produser", key: "7a7113187494906f40783d90c985d4bf" } Tue Oct 30 00:55:34 [conn88] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "534ebed7991a8f6c", key: "69b3ad1b2c773860964c40a86de5aa97" } Tue Oct 30 00:55:54 [mongosMain] connection accepted from 127.0.0.1:39286 #89 (6 connections now open) Tue Oct 30 00:56:08 [mongosMain] connection accepted from 127.0.0.1:39292 #90 (7 connections now open) Tue Oct 30 00:56:08 [conn90] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "d462dee32a351b2", user: "produser", key: "d66d27cfd455d18b77c5b7f7e71ce473" } Tue Oct 30 00:56:08 [conn90] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "87cda43faad0f8b0", key: "4d1947e0df52efa11d95a36f1e889e8e" } Tue Oct 30 00:56:43 [mongosMain] connection accepted from 127.0.0.1:39307 #91 (8 connections now open) Tue Oct 30 00:56:43 [conn91] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "f20d5fb67b25a5b4", user: "produser", key: "015154ebfbf91eb7294b9bed107e8d88" } Tue Oct 30 00:56:43 [conn91] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "cd977704f6004482", key: "b752446883c1fa94af471155590c6deb" } Tue Oct 30 00:57:18 [mongosMain] connection accepted from 127.0.0.1:39323 #92 (9 connections now open) Tue Oct 30 00:57:18 [conn92] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "5a82824f9dc12f64", user: "produser", key: "20b61cdc114fea4d0f6081e431651b66" } Tue Oct 30 00:57:18 [conn92] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "4b60cf2ee2a52212", key: "696b7b1172506459adc1587a3722cb78" } Tue Oct 30 00:57:41 [mongosMain] connection accepted from 127.0.0.1:39334 #93 (10 connections now open) Tue Oct 30 00:57:44 [conn93] end connection 127.0.0.1:39334 (9 connections now open) Tue Oct 30 00:57:53 [mongosMain] connection accepted from 127.0.0.1:39339 #94 (10 connections now open) Tue Oct 30 00:57:53 [conn94] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "1d584e461f790b49", user: "produser", key: "d02f38ced71b50b9e3b14c5eed135ad5" } Tue Oct 30 00:57:53 [conn94] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "200c47491f42eca9", key: "3b0245ab1f3c71de8066e1baae29bb8f" } Tue Oct 30 00:58:26 [CheckConfigServers] Socket recv() timeout 10.252.81.110:27019 Tue Oct 30 00:58:26 [CheckConfigServers] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] Tue Oct 30 00:58:26 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:58:26 [CheckConfigServers] warning: couldn't check on config server:c1.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c1.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:58:28 [mongosMain] connection accepted from 127.0.0.1:39353 #95 (11 connections now open) Tue Oct 30 00:58:28 [conn95] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "74ae3eddaede3034", user: "produser", key: "c5c27d76df5820e1eb73e43785b289c2" } Tue Oct 30 00:58:28 [conn95] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "31e4130c1b2cc3e3", key: "440aa7d0cc538f5fa8845af3d179c4ef" } Tue Oct 30 00:58:38 [LockPinger] Socket recv() timeout 10.252.81.110:27019 Tue Oct 30 00:58:38 [LockPinger] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [3] server [10.252.81.110:27019] Tue Oct 30 00:58:38 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:58:56 [CheckConfigServers] Socket recv() timeout 10.248.68.14:27019 Tue Oct 30 00:58:56 [CheckConfigServers] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] Tue Oct 30 00:58:56 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:58:56 [CheckConfigServers] warning: couldn't check on config server:c2.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c2.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:59:03 [mongosMain] connection accepted from 127.0.0.1:39369 #96 (12 connections now open) Tue Oct 30 00:59:03 [conn96] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "ed100db7638ea3e", user: "produser", key: "a45ed4fce370af84abd550c9950aa9ad" } Tue Oct 30 00:59:04 [conn96] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "b614270c5130203d", key: "bb76a9bcfee4e96dea660a524e731477" } Tue Oct 30 00:59:04 [mongosMain] connection accepted from 127.0.0.1:39372 #97 (13 connections now open) Tue Oct 30 00:59:08 [LockPinger] Socket recv() timeout 10.248.68.14:27019 Tue Oct 30 00:59:08 [LockPinger] SocketException: remote: 10.248.68.14:27019 error: 9001 socket exception [3] server [10.248.68.14:27019] Tue Oct 30 00:59:08 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:59:26 [CheckConfigServers] Socket recv() timeout 10.252.23.239:27019 Tue Oct 30 00:59:26 [CheckConfigServers] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] Tue Oct 30 00:59:26 [CheckConfigServers] DBClientCursor::init call() failed Tue Oct 30 00:59:26 [CheckConfigServers] warning: couldn't check on config server:c3.mongo.mydomain.com:27019 ok for now : 10276 DBClientBase::findN: transport error: c3.mongo.mydomain.com:27019 ns: admin.$cmd query: { getlasterror: 1 } Tue Oct 30 00:59:26 [CheckConfigServers] ERROR: config servers not in sync! no config servers reachable Tue Oct 30 00:59:38 [LockPinger] Socket recv() timeout 10.252.23.239:27019 Tue Oct 30 00:59:38 [LockPinger] SocketException: remote: 10.252.23.239:27019 error: 9001 socket exception [3] server [10.252.23.239:27019] Tue Oct 30 00:59:38 [LockPinger] DBClientCursor::init call() failed Tue Oct 30 00:59:38 [LockPinger] scoped connection to c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 not being returned to the pool Tue Oct 30 00:59:38 [LockPinger] warning: distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 10276 DBClientBase::findN: transport error: c1.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c1.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c2.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c2.mongo.mydomain.com:27019:{}10276 DBClientBase::findN: transport error: c3.mongo.mydomain.com:27019 ns: admin.$cmd query: { fsync: 1 } c3.mongo.mydomain.com:27019:{} Tue Oct 30 00:59:38 [mongosMain] connection accepted from 127.0.0.1:39385 #98 (14 connections now open) Tue Oct 30 00:59:38 [conn98] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "dc42eb66f1c8c021", user: "produser", key: "306806c84a9801c17519d0577ec0997f" } Tue Oct 30 00:59:38 [conn98] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "69ea9a68404e15bd", key: "4e18ae4ca06419f3765386dcfee0c171" } Tue Oct 30 01:00:08 [LockPinger] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 01:00:08 [LockPinger] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 01:00:08 [LockPinger] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 01:00:13 [mongosMain] connection accepted from 127.0.0.1:39404 #99 (15 connections now open) Tue Oct 30 01:00:13 [conn99] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "e1359bec00da92a", user: "produser", key: "0a807d1f9cb1715aeef0b42686a7cdf2" } Tue Oct 30 01:00:13 [conn99] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "add1f3d6b405521e", key: "8bd76c216c84ff496fbdf6466a136d0f" } Tue Oct 30 01:00:41 [LockPinger] cluster c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 pinged successfully at Tue Oct 30 01:00:39 2012 by distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383', sleeping for 30000ms Tue Oct 30 01:00:47 [mongosMain] connection accepted from 127.0.0.1:39422 #100 (16 connections now open) Tue Oct 30 01:00:47 [conn100] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "4df6d8991a67fa37", user: "produser", key: "bd7549abeb030630a63a51f0b0c42960" } Tue Oct 30 01:00:47 [conn100] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "7160328e8feb6523", key: "25c1f8e03a395d695d8cea818a46c27b" } Tue Oct 30 01:01:22 [mongosMain] connection accepted from 127.0.0.1:39438 #101 (17 connections now open) Tue Oct 30 01:01:22 [conn101] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "bfb88e319422e7c6", user: "produser", key: "232637c9e805549be61cca949c24ee43" } Tue Oct 30 01:01:22 [conn101] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "1910fa70ee5345e7", key: "2adefa3b6a657d9630d70ebf29c82d40" } Tue Oct 30 01:01:57 [mongosMain] connection accepted from 127.0.0.1:39452 #102 (18 connections now open) Tue Oct 30 01:01:57 [conn102] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "e9eaf957c94b2d75", user: "produser", key: "a89c394fed30559f4a50fafeed7cf824" } Tue Oct 30 01:01:57 [conn102] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "380dc13fece78654", key: "c0ca9377652c0d62fdde6a7e41ee9534" } Tue Oct 30 01:02:31 [mongosMain] connection accepted from 127.0.0.1:39467 #103 (19 connections now open) Tue Oct 30 01:02:31 [conn103] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "7219cc4ba522f8b2", user: "produser", key: "bdddb6dc9826fb119cbc3d02fe6cec1d" } Tue Oct 30 01:02:31 [conn103] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "e6323c7d371316e6", key: "278b956dc66e944c874488f01a2e125c" } Tue Oct 30 01:02:43 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:02:43 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [1] server [10.248.108.136:27018] Tue Oct 30 01:02:43 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:02:43 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n2.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:03:06 [mongosMain] connection accepted from 127.0.0.1:39483 #104 (20 connections now open) Tue Oct 30 01:03:06 [conn104] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "9fba2f04ce35e7ef", user: "produser", key: "f56db5f56e13aea7e805dda78de8541f" } Tue Oct 30 01:03:06 [conn104] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "9a19c3d2a859c7b2", key: "ee013a65550f1d798b7885b4cf6e332e" } Tue Oct 30 01:03:24 [mongosMain] connection accepted from 127.0.0.1:39492 #105 (21 connections now open) Tue Oct 30 01:03:34 [mongosMain] connection accepted from 127.0.0.1:39495 #106 (22 connections now open) Tue Oct 30 01:03:41 [mongosMain] connection accepted from 127.0.0.1:39500 #107 (23 connections now open) Tue Oct 30 01:03:41 [conn107] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "dd47cef1b3eceb7", user: "produser", key: "851c2482d25f0a0a402856275c44f7c0" } Tue Oct 30 01:03:41 [conn107] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "4fbb9e1235604182", key: "3a1685bdab085192fe439111546ca1f3" } Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.252.81.110:27018 Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [1] server [10.252.81.110:27018] Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n1.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 01:03:51 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 01:03:55 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.252.23.239:27018 Tue Oct 30 01:03:55 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [1] server [10.252.23.239:27018] Tue Oct 30 01:03:55 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:03:55 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n1.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:04:16 [mongosMain] connection accepted from 127.0.0.1:39517 #108 (24 connections now open) Tue Oct 30 01:04:16 [conn108] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "4acb1dd5c91f1d5d", user: "produser", key: "32cc028c910dfa11c4882adad0d79c51" } Tue Oct 30 01:04:16 [conn108] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "3f8b06ea357be6d6", key: "23137e38207481fd8c0452d0e76c7e77" } Tue Oct 30 01:04:51 [mongosMain] connection accepted from 127.0.0.1:39532 #109 (25 connections now open) Tue Oct 30 01:04:51 [conn109] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "9faea5e0c2f9ec27", user: "produser", key: "6e8a05279a604bab2faca2d2215ec4b3" } Tue Oct 30 01:04:51 [conn109] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "e28ee1a2e2ec7e08", key: "3b4485f8fbb32524df52ce5054b98d98" } Tue Oct 30 01:05:25 [mongosMain] connection accepted from 127.0.0.1:39548 #110 (26 connections now open) Tue Oct 30 01:05:25 [conn110] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "b3e2de15849a0e82", user: "produser", key: "14a58e27c8eb2e8abedd7241bb890a63" } Tue Oct 30 01:05:25 [conn110] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "fc7a27a73e03009f", key: "28998baf326b47a6bdd1fc8a309282ef" } Tue Oct 30 01:05:45 [LockPinger] cluster c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 pinged successfully at Tue Oct 30 01:05:45 2012 by distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383', sleeping for 30000ms Tue Oct 30 01:06:00 [mongosMain] connection accepted from 127.0.0.1:39564 #111 (27 connections now open) Tue Oct 30 01:06:00 [conn111] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "d5d5d5a29013c1ee", user: "produser", key: "bda63068c89747bd1fc0a83ba3fba5a4" } Tue Oct 30 01:06:00 [conn111] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "a405d94465b80f23", key: "8ea9f342a1a139851f349f88f12e5978" } Tue Oct 30 01:06:21 [WriteBackListener-r1.n2.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.248.68.14:27018 Tue Oct 30 01:06:21 [WriteBackListener-r1.n2.mongo.mydomain.com:27018] SocketException: remote: 10.248.68.14:27018 error: 9001 socket exception [1] server [10.248.68.14:27018] Tue Oct 30 01:06:21 [WriteBackListener-r1.n2.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:06:21 [WriteBackListener-r1.n2.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n2.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:06:34 [mongosMain] connection accepted from 127.0.0.1:39577 #112 (28 connections now open) Tue Oct 30 01:06:34 [conn112] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "d5a7615b738d584c", user: "produser", key: "d4530d46a5d985ff286e97cfa834abe5" } Tue Oct 30 01:06:34 [conn112] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "6739bdf69c2f65bd", key: "f068a28d01d842cdb4a741916cfa2891" } Tue Oct 30 01:06:34 [ReplicaSetMonitorWatcher] Socket recv() errno:110 Connection timed out 10.252.81.110:27018 Tue Oct 30 01:06:34 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [1] server [10.252.81.110:27018] Tue Oct 30 01:06:34 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 01:06:39 [Balancer] Socket recv() errno:110 Connection timed out 10.252.81.110:27019 Tue Oct 30 01:06:39 [Balancer] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [1] server [10.252.81.110:27019] Tue Oct 30 01:06:39 [Balancer] DBClientCursor::init call() failed Tue Oct 30 01:06:39 [Balancer] query failed to: c1.mongo.mydomain.com:27019 no data Tue Oct 30 01:06:39 [ReplicaSetMonitorWatcher] Socket recv() timeout 10.252.81.110:27018 Tue Oct 30 01:06:39 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [3] server [10.252.81.110:27018] Tue Oct 30 01:06:39 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 01:06:44 [ReplicaSetMonitorWatcher] Socket recv() timeout 10.248.68.14:27018 Tue Oct 30 01:06:44 [ReplicaSetMonitorWatcher] SocketException: remote: 10.248.68.14:27018 error: 9001 socket exception [3] server [10.248.68.14:27018] Tue Oct 30 01:06:44 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 01:06:44 [conn87] trying reconnect to r1.n1.mongo.mydomain.com:27018 Tue Oct 30 01:06:45 [conn87] reconnect r1.n1.mongo.mydomain.com:27018 ok Tue Oct 30 01:06:45 [conn87] Socket say send() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:06:45 [conn87] DBException in process: socket exception [SEND_ERROR] for 10.248.108.136:27018 Tue Oct 30 01:06:45 [conn87] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39259 Tue Oct 30 01:06:45 [conn87] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39259] Tue Oct 30 01:06:45 [conn89] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39286 Tue Oct 30 01:06:45 [conn89] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39286] Tue Oct 30 01:06:45 [conn88] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39277 Tue Oct 30 01:06:45 [conn88] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39277] Tue Oct 30 01:06:45 [conn94] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39339 Tue Oct 30 01:06:45 [conn94] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39339] Tue Oct 30 01:06:45 [conn91] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39307 Tue Oct 30 01:06:45 [conn91] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39307] Tue Oct 30 01:06:45 [conn98] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39385 Tue Oct 30 01:06:45 [conn98] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39385] Tue Oct 30 01:06:45 [conn90] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39292 Tue Oct 30 01:06:45 [conn90] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39292] Tue Oct 30 01:06:45 [conn96] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39369 Tue Oct 30 01:06:45 [conn96] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39369] Tue Oct 30 01:06:45 [conn99] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39404 Tue Oct 30 01:06:45 [conn99] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39404] Tue Oct 30 01:06:45 [conn95] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39353 Tue Oct 30 01:06:45 [conn95] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39353] Tue Oct 30 01:06:45 [conn92] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39323 Tue Oct 30 01:06:45 [conn92] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39323] Tue Oct 30 01:06:45 [conn85] Socket say send() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:06:45 [conn85] DBException in process: socket exception [SEND_ERROR] for 10.248.108.136:27018 Tue Oct 30 01:06:45 [conn85] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39145 Tue Oct 30 01:06:45 [conn85] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39145] Tue Oct 30 01:06:45 [conn101] end connection 127.0.0.1:39438 (15 connections now open) Tue Oct 30 01:06:45 [conn103] end connection 127.0.0.1:39467 (15 connections now open) Tue Oct 30 01:06:45 [conn100] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:39422 Tue Oct 30 01:06:45 [conn100] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:39422] Tue Oct 30 01:06:45 [conn104] end connection 127.0.0.1:39483 (12 connections now open) Tue Oct 30 01:06:45 [conn109] end connection 127.0.0.1:39532 (12 connections now open) Tue Oct 30 01:06:45 [conn108] end connection 127.0.0.1:39517 (12 connections now open) Tue Oct 30 01:06:45 [conn105] Socket say send() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:06:45 [conn105] end connection 127.0.0.1:39492 (9 connections now open) Tue Oct 30 01:06:45 [conn110] end connection 127.0.0.1:39548 (10 connections now open) Tue Oct 30 01:06:45 [conn107] end connection 127.0.0.1:39500 (7 connections now open) Tue Oct 30 01:06:45 [conn106] end connection 127.0.0.1:39495 (6 connections now open) Tue Oct 30 01:06:45 [conn102] end connection 127.0.0.1:39452 (5 connections now open) Tue Oct 30 01:06:45 [conn111] end connection 127.0.0.1:39564 (4 connections now open) Tue Oct 30 01:06:49 [ReplicaSetMonitorWatcher] Socket recv() timeout 10.248.108.136:27018 Tue Oct 30 01:06:49 [ReplicaSetMonitorWatcher] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [3] server [10.248.108.136:27018] Tue Oct 30 01:06:49 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 01:06:54 [ReplicaSetMonitorWatcher] Socket recv() timeout 10.252.23.239:27018 Tue Oct 30 01:06:54 [ReplicaSetMonitorWatcher] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [3] server [10.252.23.239:27018] Tue Oct 30 01:06:54 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 30 01:06:54 [ReplicaSetMonitorWatcher] trying reconnect to r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:06:55 [ReplicaSetMonitorWatcher] reconnect r2.n2.mongo.mydomain.com:27018 ok Tue Oct 30 01:06:55 [ReplicaSetMonitorWatcher] Socket say send() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:07:05 [ReplicaSetMonitorWatcher] trying reconnect to r1.n2.mongo.mydomain.com:27018 Tue Oct 30 01:07:05 [ReplicaSetMonitorWatcher] reconnect r1.n2.mongo.mydomain.com:27018 ok Tue Oct 30 01:07:05 [ReplicaSetMonitorWatcher] trying reconnect to r2.n1.mongo.mydomain.com:27018 Tue Oct 30 01:07:05 [ReplicaSetMonitorWatcher] reconnect r2.n1.mongo.mydomain.com:27018 ok Tue Oct 30 01:07:08 [mongosMain] connection accepted from 127.0.0.1:39635 #113 (5 connections now open) Tue Oct 30 01:07:08 [conn113] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, nonce: "6e87989cc6793078", user: "produser", key: "76d4ea8d563ad71a79439352e05b27ef" } Tue Oct 30 01:07:08 [conn113] authenticate db: myapp2-prod-clustered_interests { authenticate: 1, user: "produser", nonce: "e2a598f783ca23e7", key: "68f6d281b56403bd02d0571765352e14" } Tue Oct 30 01:09:03 [conn84] Socket recv() errno:110 Connection timed out 10.248.108.136:27018 Tue Oct 30 01:09:03 [conn84] SocketException: remote: 10.248.108.136:27018 error: 9001 socket exception [1] server [10.248.108.136:27018] Tue Oct 30 01:09:03 [conn84] DBClientCursor::init lazy say() failed Tue Oct 30 01:09:03 [conn84] DBClientCursor::init message from say() was empty Tue Oct 30 01:09:03 [conn84] got not master for: r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:09:03 [conn84] warning: db exception when finishing on replSet2:replSet2/r2.n1.mongo.mydomain.com:27018,r2.n2.mongo.mydomain.com:27018, current connection state is { state: { conn: "replSet2/r2.n1.mongo.mydomain.com:27018,r2.n2.mongo.mydomain.com:27018", vinfo: "replSet2:replSet2/r2.n1.mongo.mydomain.com:27018,r2.n2.mongo.mydomain.com:27018", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server Tue Oct 30 01:09:03 [conn84] DBException in process: socket exception [SEND_ERROR] for 127.0.0.1:38511 Tue Oct 30 01:09:03 [conn84] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:38511] Tue Oct 30 01:09:05 [ReplicaSetMonitorWatcher] Primary for replica set replSet2 changed to r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:10:48 [LockPinger] cluster c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 pinged successfully at Tue Oct 30 01:10:47 2012 by distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383', sleeping for 30000ms Tue Oct 30 01:12:43 [mongosMain] connection accepted from 127.0.0.1:39710 #114 (5 connections now open) Tue Oct 30 01:14:11 [conn114] end connection 127.0.0.1:39710 (4 connections now open) Tue Oct 30 01:15:50 [LockPinger] cluster c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 pinged successfully at Tue Oct 30 01:15:50 2012 by distributed lock pinger 'c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019/ip-10-252-160-159:27017:1350954554:1804289383', sleeping for 30000ms Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.252.81.110:27019 Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] SocketException: remote: 10.252.81.110:27019 error: 9001 socket exception [1] server [10.252.81.110:27019] Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] query failed to: c1.mongo.mydomain.com:27019 no data Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] Socket say send() errno:110 Connection timed out 10.248.68.14:27019 Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] query failed to: c2.mongo.mydomain.com:27019 exception: socket exception [SEND_ERROR] for 10.248.68.14:27019 Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] Socket say send() errno:110 Connection timed out 10.252.23.239:27019 Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] query failed to: c3.mongo.mydomain.com:27019 exception: socket exception [SEND_ERROR] for 10.252.23.239:27019 Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] scoped connection to c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 not being returned to the pool Tue Oct 30 01:18:22 [WriteBackListener-r2.n2.mongo.mydomain.com:27018] ERROR: backgroundjob WriteBackListener-r2.n2.mongo.mydomain.com:27018error: all servers down! Tue Oct 30 01:19:24 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.252.81.110:27018 Tue Oct 30 01:19:24 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] SocketException: remote: 10.252.81.110:27018 error: 9001 socket exception [1] server [10.252.81.110:27018] Tue Oct 30 01:19:24 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:19:24 [WriteBackListener-r1.n1.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r1.n1.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:19:27 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] Socket recv() errno:110 Connection timed out 10.252.23.239:27018 Tue Oct 30 01:19:27 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] SocketException: remote: 10.252.23.239:27018 error: 9001 socket exception [1] server [10.252.23.239:27018] Tue Oct 30 01:19:27 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] DBClientCursor::init call() failed Tue Oct 30 01:19:27 [WriteBackListener-r2.n1.mongo.mydomain.com:27018] WriteBackListener exception : DBClientBase::findN: transport error: r2.n1.mongo.mydomain.com:27018 ns: admin.$cmd query: { writebacklisten: ObjectId('5085ee3844e086d5bb061d4d') } Tue Oct 30 01:20:49 [mongosMain] dbexit: received signal 15 rc:0 received signal 15 ***** SERVER RESTARTED ***** Tue Oct 30 01:20:49 [mongosMain] MongoS version 2.2.0 starting: pid=8154 port=27017 64-bit host=ip-10-252-160-159 (--help for usage) Tue Oct 30 01:20:49 [mongosMain] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207 Tue Oct 30 01:20:49 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49 Tue Oct 30 01:20:49 [mongosMain] options: { config: "/etc/mongos.conf", configdb: "c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019", logappend: "true", logpath: "/var/log/mongodb/mongos.log" } Tue Oct 30 01:20:49 [mongosMain] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [mongosMain] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [mongosMain] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [mongosMain] waiting for connections on port 27017 Tue Oct 30 01:20:49 [Balancer] about to contact config servers and shards Tue Oct 30 01:20:49 [Balancer] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [websvr] admin web console waiting for connections on port 28017 Tue Oct 30 01:20:49 [Balancer] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [Balancer] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 01:20:49 [Balancer] starting new replica set monitor for replica set replSet1 with seed of r1.n1.mongo.mydomain.com:27018,r1.n2.mongo.mydomain.com:27018 Tue Oct 30 01:20:49 [Balancer] successfully connected to seed r1.n1.mongo.mydomain.com:27018 for replica set replSet1 Tue Oct 30 01:20:49 [Balancer] changing hosts to { 0: "r1.n1.mongo.mydomain.com:27018", 1: "r1.n2.mongo.mydomain.com:27018" } from replSet1/ Tue Oct 30 01:20:49 [Balancer] trying to add new host r1.n1.mongo.mydomain.com:27018 to replica set replSet1 Tue Oct 30 01:20:49 [Balancer] successfully connected to new host r1.n1.mongo.mydomain.com:27018 in replica set replSet1 Tue Oct 30 01:20:49 [Balancer] trying to add new host r1.n2.mongo.mydomain.com:27018 to replica set replSet1 Tue Oct 30 01:20:49 [Balancer] successfully connected to new host r1.n2.mongo.mydomain.com:27018 in replica set replSet1 Tue Oct 30 01:20:49 [Balancer] Primary for replica set replSet1 changed to r1.n1.mongo.mydomain.com:27018 Tue Oct 30 01:20:49 [Balancer] replica set monitor for replica set replSet1 started, address is replSet1/r1.n1.mongo.mydomain.com:27018,r1.n2.mongo.mydomain.com:27018 Tue Oct 30 01:20:49 [ReplicaSetMonitorWatcher] starting Tue Oct 30 01:20:49 [Balancer] starting new replica set monitor for replica set replSet2 with seed of r2.n1.mongo.mydomain.com:27018,r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:20:49 [Balancer] successfully connected to seed r2.n1.mongo.mydomain.com:27018 for replica set replSet2 Tue Oct 30 01:20:49 [Balancer] changing hosts to { 0: "r2.n1.mongo.mydomain.com:27018", 1: "r2.n2.mongo.mydomain.com:27018" } from replSet2/ Tue Oct 30 01:20:49 [Balancer] trying to add new host r2.n1.mongo.mydomain.com:27018 to replica set replSet2 Tue Oct 30 01:20:49 [Balancer] successfully connected to new host r2.n1.mongo.mydomain.com:27018 in replica set replSet2 Tue Oct 30 01:20:49 [Balancer] trying to add new host r2.n2.mongo.mydomain.com:27018 to replica set replSet2 Tue Oct 30 01:20:49 [Balancer] successfully connected to new host r2.n2.mongo.mydomain.com:27018 in replica set replSet2 Tue Oct 30 01:20:50 [Balancer] Primary for replica set replSet2 changed to r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:20:50 [Balancer] replica set monitor for replica set replSet2 started, address is replSet2/r2.n1.mongo.mydomain.com:27018,r2.n2.mongo.mydomain.com:27018 Tue Oct 30 01:20:50 [Balancer] config servers and shards contacted successfully Tue Oct 30 01:20:50 [Balancer] balancer id: ip-10-252-160-159:27017 started at Oct 30 01:20:50 Tue Oct 30 01:20:50 [Balancer] created new distributed lock for balancer on c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 ( lock timeout : 900000, ping interval : 30000, process : 0 ) Tue Oct 30 01:20:50 [Balancer] SyncClusterConnection connecting to [c1.mongo.mydomain.com:27019] Tue Oct 30 01:20:50 [Balancer] SyncClusterConnection connecting to [c2.mongo.mydomain.com:27019] Tue Oct 30 01:20:50 [Balancer] SyncClusterConnection connecting to [c3.mongo.mydomain.com:27019] Tue Oct 30 01:20:50 [LockPinger] creating distributed lock ping thread for c1.mongo.mydomain.com:27019,c2.mongo.mydomain.com:27019,c3.mongo.mydomain.com:27019 and process ip-10-252-160-159:27017:1351560050:1804289383 (sleeping for 30000ms) Tue Oct 30 01:20:50 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b725a9ac1f41690105f Tue Oct 30 01:20:51 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:20:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b795a9ac1f416901060 Tue Oct 30 01:20:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b7f5a9ac1f416901061 Tue Oct 30 01:21:04 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b865a9ac1f416901062 Tue Oct 30 01:21:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b8c5a9ac1f416901063 Tue Oct 30 01:21:17 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b935a9ac1f416901064 Tue Oct 30 01:21:24 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:30 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2b9a5a9ac1f416901065 Tue Oct 30 01:21:31 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2ba15a9ac1f416901066 Tue Oct 30 01:21:37 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:44 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2ba75a9ac1f416901067 Tue Oct 30 01:21:44 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:21:50 [Balancer] could not acquire lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' (another update won) Tue Oct 30 01:21:50 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' was not acquired. Tue Oct 30 01:21:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bb45a9ac1f416901069 Tue Oct 30 01:21:57 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bc15a9ac1f41690106a Tue Oct 30 01:22:10 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:23 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bce5a9ac1f41690106b Tue Oct 30 01:22:23 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:36 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bdb5a9ac1f41690106c Tue Oct 30 01:22:36 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:42 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2be25a9ac1f41690106d Tue Oct 30 01:22:43 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:49 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2be95a9ac1f41690106e Tue Oct 30 01:22:49 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:22:55 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bef5a9ac1f41690106f Tue Oct 30 01:22:55 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:02 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bf55a9ac1f416901070 Tue Oct 30 01:23:02 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:08 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2bfc5a9ac1f416901071 Tue Oct 30 01:23:09 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:15 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c035a9ac1f416901072 Tue Oct 30 01:23:15 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:21 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c095a9ac1f416901073 Tue Oct 30 01:23:22 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:28 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c105a9ac1f416901074 Tue Oct 30 01:23:28 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:34 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c165a9ac1f416901075 Tue Oct 30 01:23:35 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:41 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c1d5a9ac1f416901076 Tue Oct 30 01:23:41 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:47 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c235a9ac1f416901077 Tue Oct 30 01:23:47 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:23:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c295a9ac1f416901078 Tue Oct 30 01:23:54 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:24:07 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c365a9ac1f416901079 Tue Oct 30 01:24:07 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:24:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c495a9ac1f41690107a Tue Oct 30 01:24:26 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:24:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c505a9ac1f41690107b Tue Oct 30 01:24:32 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:24:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c565a9ac1f41690107c Tue Oct 30 01:24:39 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' unlocked. Tue Oct 30 01:24:51 [Balancer] distributed lock 'balancer/ip-10-252-160-159:27017:1351560050:1804289383' acquired, ts : 508f2c635a9ac1f41690107d