Sun Apr 7 04:00:11 [rsHealthPoll] replSet member crwlrdb04:8086 is now in state RECOVERING
|
Sun Apr 7 04:00:12 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:00:12 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: crwlrdb04:8086
|
Sun Apr 7 04:00:15 [rsHealthPoll] replSet member crwlrdb04:8086 is now in state SECONDARY
|
Sun Apr 7 04:00:16 [rsMgr] replSet info electSelf 1
|
Sun Apr 7 04:00:16 [rsMgr] replSet PRIMARY
|
Sun Apr 7 04:00:17 [conn913928] query local.oplog.rs query: { ts: { $gte: new Date(5864049901565902853) } } cursorid:3635176764573440955 ntoreturn:0 ntoskip:0
|
nscanned:2 keyUpdates:0 locks(micros) r:104450 nreturned:1 reslen:217 104ms
|
Sun Apr 7 04:00:25 [conn913936] no current chunk manager found for this shard, will initialize
|
Sun Apr 7 04:00:25 [conn913936] setShardVersion - relocking slow: 96
|
...
|
Sun Apr 7 04:10:17 [conn914483] replSet info stepping down as primary secs=1
|
Sun Apr 7 04:10:17 [conn914483] replSet relinquishing primary state
|
Sun Apr 7 04:10:17 [conn914483] replSet SECONDARY
|
Sun Apr 7 04:10:17 [conn914483] replSet closing client sockets after relinquishing primary
|
Sun Apr 7 04:10:18 [rsHealthPoll] replSet info crwlrdb06:8086 is down (or slow to respond): socket exception [SEND_ERROR] for 10.101.0.72:8086
|
Sun Apr 7 04:10:18 [rsHealthPoll] replSet member crwlrdb06:8086 is now in state DOWN
|
Sun Apr 7 04:10:18 [rsMgr] replSet info electSelf 1
|
Sun Apr 7 04:10:18 [rsMgr] replSet couldn't elect self, only received -19999 votes
|
Sun Apr 7 04:10:19 [conn913928] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.101.0.70:38568]
|
Sun Apr 7 04:10:20 [rsHealthPoll] replSet member crwlrdb06:8086 is up
|
Sun Apr 7 04:10:20 [rsHealthPoll] replSet member crwlrdb06:8086 is now in state ARBITER
|
Sun Apr 7 04:10:20 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:20 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 10.101.0.25:8086
|
Sun Apr 7 04:10:20 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 10.101.0.24:8086
|
Sun Apr 7 04:10:20 [ReplicaSetMonitorWatcher] trying reconnect to crwlrdb02:8086
|
Sun Apr 7 04:10:20 [ReplicaSetMonitorWatcher] reconnect crwlrdb02:8086 ok
|
Sun Apr 7 04:10:20 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 10.101.0.25:8086
|
Sun Apr 7 04:10:22 [conn914484] replSet info voting yea for crwlrdb04:8086 (0)
|
Sun Apr 7 04:10:26 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:30 [ReplicaSetMonitorWatcher] trying reconnect to crwlrdb01:8086
|
Sun Apr 7 04:10:30 [ReplicaSetMonitorWatcher] reconnect crwlrdb01:8086 ok
|
Sun Apr 7 04:10:30 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 10.101.0.24:8086
|
Sun Apr 7 04:10:32 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:37 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:38 [LockPinger] Socket say send() errno:9 Bad file descriptor 10.101.0.43:8086
|
Sun Apr 7 04:10:38 [LockPinger] Socket say send() errno:9 Bad file descriptor 10.101.0.44:8086
|
Sun Apr 7 04:10:38 [LockPinger] Socket say send() errno:9 Bad file descriptor 10.101.0.45:8086
|
Sun Apr 7 04:10:38 [LockPinger] scoped connection to crwlrcfgdb01:8086,crwlrcfgdb02:8086,crwlrcfgdb03:8086 not being returned to the pool
|
Sun Apr 7 04:10:38 [LockPinger] warning: distributed lock pinger 'crwlrcfgdb01:8086,crwlrcfgdb02:8086,crwlrcfgdb03:8086/crwlrdb05.prod.dal.us:8086:1361707225:275889730' detected an exception while pinging. :: caused by :: SyncClusterConnection::udpate prepare failed: 9001 socket exception [2] server [10.101.0.43:8086] crwlrcfgdb01:8086:{}9001 socket exception [2] server [10.101.0.44:8086] crwlrcfgdb02:8086:{}9001 socket exception [2] server [10.101.0.45:8086] crwlrcfgdb03:8086:{}
|
Sun Apr 7 04:10:42 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:47 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:52 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:10:57 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:11:02 [rsMgr] not electing self, backup.ec2:18089 would veto
|
Sun Apr 7 04:11:07 [rsMgr] not electing self, backup.ec2:18089 would veto
|
...
|