2016-11-02T14:10:45.383-0400 I CONTROL [main] 2016-11-02T14:10:45.383-0400 I CONTROL [main] ** WARNING: Access control is not enabled for the database. 2016-11-02T14:10:45.383-0400 I CONTROL [main] ** Read and write access to data and configuration is unrestricted. 2016-11-02T14:10:45.383-0400 I CONTROL [main] 2016-11-02T14:10:45.383-0400 I SHARDING [mongosMain] mongos version v3.4.0-rc2 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] git version: 217fb0a4d2b808fa45921e4ca1d73db32330620b 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] allocator: system 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] modules: none 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] build environment: 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] distarch: x86_64 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] target_arch: x86_64 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] db version v3.4.0-rc2 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] git version: 217fb0a4d2b808fa45921e4ca1d73db32330620b 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] allocator: system 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] modules: none 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] build environment: 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] distarch: x86_64 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] target_arch: x86_64 2016-11-02T14:10:45.383-0400 I CONTROL [mongosMain] options: { sharding: { configDB: "csrs/localhost:27002" }, systemLog: { destination: "file", path: "mongos.log" } } 2016-11-02T14:10:45.392-0400 I NETWORK [mongosMain] Starting new replica set monitor for csrs/localhost:27002 2016-11-02T14:10:45.392-0400 I SHARDING [thread1] creating distributed lock ping thread for process liquid.local:27017:1478110245:2790712912708122139 (sleeping for 30000ms) 2016-11-02T14:10:45.393-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] changing hosts to csrs/liquid.local:27002 from csrs/localhost:27002 2016-11-02T14:10:45.393-0400 I SHARDING [ReplicaSetMonitor-TaskExecutor-0] Updating ShardRegistry connection string for shard config from: csrs/localhost:27002 to: csrs/liquid.local:27002 2016-11-02T14:10:45.394-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:10:45.394-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:10:45.394-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:10:45.396-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:10:45.396-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:10:45.397-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:10:45.397-0400 I NETWORK [thread2] waiting for connections on port 27017 2016-11-02T14:10:45.405-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document 2016-11-02T14:11:06.410-0400 I NETWORK [thread2] connection accepted from 127.0.0.1:59097 #1 (1 connection now open) 2016-11-02T14:11:06.410-0400 I NETWORK [conn1] received client metadata from 127.0.0.1:59097 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:15.510-0400 I NETWORK [conn1] Starting new replica set monitor for rs0/liquid.local:27000 2016-11-02T14:11:20.173-0400 I NETWORK [conn1] Starting new replica set monitor for rs1/liquid.local:27001 2016-11-02T14:11:23.858-0400 I SHARDING [conn1] distributed lock 'testDb' acquired for 'enableSharding', ts : 581a2c4b172517c21c78f87c 2016-11-02T14:11:23.858-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27000 2016-11-02T14:11:23.860-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27000 2016-11-02T14:11:23.862-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27001 2016-11-02T14:11:23.863-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27001 2016-11-02T14:11:23.863-0400 I SHARDING [conn1] Placing [testDb] on: rs0 2016-11-02T14:11:23.863-0400 I SHARDING [conn1] Enabling sharding for database [testDb] in config db 2016-11-02T14:11:23.922-0400 I SHARDING [conn1] distributed lock with ts: 581a2c4b172517c21c78f87c' unlocked. 2016-11-02T14:11:39.648-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to liquid.local:27000 2016-11-02T14:11:39.650-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to liquid.local:27000 2016-11-02T14:11:57.098-0400 I COMMAND [conn1] CMD: shardcollection: { shardCollection: "testDb.testColl", key: { b: 1.0, c: 1.0 } } 2016-11-02T14:11:57.105-0400 I SHARDING [conn1] distributed lock 'testDb.testColl' acquired for 'shardCollection', ts : 581a2c6d172517c21c78fc6e 2016-11-02T14:11:57.108-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "liquid.local-2016-11-02T14:11:57.107-0400-581a2c6d172517c21c78fc6f", server: "liquid.local", clientAddr: "127.0.0.1:59097", time: new Date(1478110317107), what: "shardCollection.start", ns: "testDb.testColl", details: { shardKey: { b: 1.0, c: 1.0 }, collection: "testDb.testColl", primary: "rs0:rs0/liquid.local:27000", initShards: [], numChunks: 1 } } 2016-11-02T14:11:57.115-0400 I SHARDING [conn1] going to create 1 chunk(s) for: testDb.testColl using new epoch 581a2c6d172517c21c78fc70 2016-11-02T14:11:57.123-0400 I SHARDING [conn1] ChunkManager loading chunks for testDb.testColl sequenceNumber: 2 based on: (empty) 2016-11-02T14:11:57.123-0400 I SHARDING [conn1] ChunkManager load took 0 ms and found version 1|0||581a2c6d172517c21c78fc70 2016-11-02T14:11:57.184-0400 I SHARDING [conn1] about to log metadata event into changelog: { _id: "liquid.local-2016-11-02T14:11:57.184-0400-581a2c6d172517c21c78fc71", server: "liquid.local", clientAddr: "127.0.0.1:59097", time: new Date(1478110317184), what: "shardCollection.end", ns: "testDb.testColl", details: { version: "1|0||581a2c6d172517c21c78fc70" } } 2016-11-02T14:11:57.205-0400 I SHARDING [conn1] distributed lock with ts: 581a2c6d172517c21c78fc6e' unlocked. 2016-11-02T14:11:57.205-0400 I SHARDING [conn1] ChunkManager loading chunks for testDb.testColl sequenceNumber: 3 based on: (empty) 2016-11-02T14:11:57.206-0400 I SHARDING [conn1] ChunkManager load took 0 ms and found version 1|0||581a2c6d172517c21c78fc70 2016-11-02T14:12:41.245-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Connecting to liquid.local:27002 2016-11-02T14:12:41.247-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to liquid.local:27002 2016-11-02T14:12:45.397-0400 I NETWORK [PeriodicTaskRunner] Socket closed remotely, no longer connected (idle 30 secs, remote host 10.0.1.3:27002) 2016-11-02T14:12:45.397-0400 I NETWORK [PeriodicTaskRunner] Socket closed remotely, no longer connected (idle 60 secs, remote host 127.0.0.1:27002) 2016-11-02T14:12:45.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.398-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Marking host liquid.local:27002 as failed :: caused by :: Location13328: connection pool: connect failed liquid.local:27002 : couldn't connect to server liquid.local:27002, connection attempt failed 2016-11-02T14:12:45.398-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:12:45.398-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] All nodes for set csrs are down. This has happened for 1 checks in a row. 2016-11-02T14:12:45.398-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.398-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:45.398-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 2 checks in a row. 2016-11-02T14:12:45.399-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.399-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to close stream: Socket is not connected 2016-11-02T14:12:45.399-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:45.399-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 3 checks in a row. 2016-11-02T14:12:45.464-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.464-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:45.464-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 4 checks in a row. 2016-11-02T14:12:45.597-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.598-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:45.598-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 5 checks in a row. 2016-11-02T14:12:45.900-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.900-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:45.900-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 6 checks in a row. 2016-11-02T14:12:45.965-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.965-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:45.965-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 7 checks in a row. 2016-11-02T14:12:46.100-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.100-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:46.100-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 8 checks in a row. 2016-11-02T14:12:46.401-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.401-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:46.401-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 9 checks in a row. 2016-11-02T14:12:46.468-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.468-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:46.468-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 10 checks in a row. 2016-11-02T14:12:46.601-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.601-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:46.601-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 11 checks in a row. 2016-11-02T14:12:46.907-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.907-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:46.973-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.973-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:47.106-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.107-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:47.408-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.408-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:47.475-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.475-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:47.581-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.581-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:47.608-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.608-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:47.909-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.909-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:47.976-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.976-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:48.082-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.082-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:48.082-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 21 checks in a row. 2016-11-02T14:12:48.109-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.109-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:48.410-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.411-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:48.477-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.477-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:48.583-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.583-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:48.614-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.614-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:48.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:48.979-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.979-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:49.086-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.086-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:49.119-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.119-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:49.413-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.413-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:49.413-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 31 checks in a row. 2016-11-02T14:12:49.482-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.483-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:49.591-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.591-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:49.623-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.623-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:49.915-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.916-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:49.986-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.986-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:50.092-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.092-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:50.128-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.128-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:50.418-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.418-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:50.487-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.487-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:50.594-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.594-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:50.594-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 41 checks in a row. 2016-11-02T14:12:50.630-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.630-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:50.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.924-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:50.988-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.989-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.095-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.095-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:51.136-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.136-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:51.425-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.425-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.489-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.489-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.599-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.599-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:51.639-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.639-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:51.926-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.927-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.927-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 51 checks in a row. 2016-11-02T14:12:51.990-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.990-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:52.100-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.100-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:52.142-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.142-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:52.428-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.428-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.491-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.491-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:52.602-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.602-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:52.645-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.645-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:52.932-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.933-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.993-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.993-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.103-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.103-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:53.103-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 61 checks in a row. 2016-11-02T14:12:53.146-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.146-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:53.433-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.434-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.496-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.496-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.604-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.604-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:53.646-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.646-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:53.935-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.935-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.996-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.996-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.105-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.105-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:54.147-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.147-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:54.439-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.439-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:54.439-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 71 checks in a row. 2016-11-02T14:12:54.499-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.500-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.610-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.610-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:54.651-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.651-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:54.944-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.945-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.002-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.002-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:55.111-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.111-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:55.152-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.152-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:55.446-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.446-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.503-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.503-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:55.612-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.612-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:55.612-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 81 checks in a row. 2016-11-02T14:12:55.653-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.653-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:55.947-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.948-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.005-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.005-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.114-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.114-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:56.154-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.154-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:56.449-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.450-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.507-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.507-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.616-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.616-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:56.659-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.659-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:56.955-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.955-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.955-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 91 checks in a row. 2016-11-02T14:12:57.010-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.010-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.117-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.117-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:57.163-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.163-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:57.459-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.459-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:57.513-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.513-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.620-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.620-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:57.664-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.664-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:57.961-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.961-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.015-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.015-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.124-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.124-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:58.124-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 101 checks in a row. 2016-11-02T14:12:58.165-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.165-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:58.463-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.463-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.516-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.516-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.626-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.626-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:58.670-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.670-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:58.969-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.969-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:59.017-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.018-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.127-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.127-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:59.171-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.171-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:59.473-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.473-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:12:59.473-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 111 checks in a row. 2016-11-02T14:12:59.518-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.519-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.629-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.629-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:12:59.672-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.672-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:12:59.978-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.978-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:00.020-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.020-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:00.130-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.131-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:00.177-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.177-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:00.481-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.481-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:00.522-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.522-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:00.631-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.632-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:00.632-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 121 checks in a row. 2016-11-02T14:13:00.680-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.680-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:00.984-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.984-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.025-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.025-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.136-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.136-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:01.183-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.183-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:01.485-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.485-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.529-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.529-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.640-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.640-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:01.686-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.686-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:01.987-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.987-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.987-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 131 checks in a row. 2016-11-02T14:13:02.034-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.034-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.144-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.144-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:02.190-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.190-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:02.492-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.492-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:02.535-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.535-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.649-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.649-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:02.694-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.694-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:02.994-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.994-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:03.039-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.039-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:03.152-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.153-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:03.153-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 141 checks in a row. 2016-11-02T14:13:03.195-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.195-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:03.498-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.498-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:03.540-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.540-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:03.656-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.656-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:03.696-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.696-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:04.002-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.002-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:04.041-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.041-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.157-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.157-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:04.197-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.197-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:04.503-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.504-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:04.504-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 151 checks in a row. 2016-11-02T14:13:04.542-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.542-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.658-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.658-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:04.702-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.702-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:05.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.010-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:05.043-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.043-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.160-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.161-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:05.204-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.204-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:05.512-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.512-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:05.512-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:13:05.512-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:13:05.544-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.544-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.544-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:13:05.664-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.664-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:05.664-0400 I NETWORK [conn1] All nodes for set csrs are down. This has happened for 161 checks in a row. 2016-11-02T14:13:05.705-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.705-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:05.706-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.706-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:06.166-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.166-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:06.209-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.209-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:06.667-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.667-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:06.714-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.714-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:07.168-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.168-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:07.215-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.215-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:07.527-0400 I NETWORK [thread2] connection accepted from 127.0.0.1:59523 #2 (2 connections now open) 2016-11-02T14:13:07.527-0400 I NETWORK [conn2] received client metadata from 127.0.0.1:59523 conn2: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:13:07.528-0400 I - [conn2] end connection 127.0.0.1:59523 (2 connections now open) 2016-11-02T14:13:07.669-0400 W NETWORK [conn1] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.669-0400 W NETWORK [conn1] No primary detected for set csrs 2016-11-02T14:13:07.669-0400 I - [conn1] end connection 127.0.0.1:59097 (1 connection now open) 2016-11-02T14:13:07.719-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.720-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:07.720-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 171 checks in a row. 2016-11-02T14:13:08.224-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.224-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:08.729-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.729-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:09.231-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.231-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:09.733-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.733-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:10.238-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:10.238-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:10.739-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:10.739-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:11.243-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:11.243-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:11.747-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:11.747-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:12.252-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:12.252-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:12.755-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:12.755-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:12.755-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 181 checks in a row. 2016-11-02T14:13:13.257-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:13.257-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:13.761-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:13.761-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:14.262-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:14.262-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:14.763-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:14.763-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:15.268-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:15.268-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:15.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:15.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:15.771-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:15.771-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:15.904-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:15.904-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:16.276-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:16.276-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:16.409-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:16.409-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:16.409-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 191 checks in a row. 2016-11-02T14:13:16.780-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:16.780-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:16.914-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:16.914-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:17.282-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:17.282-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:17.419-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:17.419-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:17.785-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:17.785-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:17.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:17.924-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:18.290-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:18.290-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:18.428-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:18.428-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:18.794-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:18.795-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:18.931-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:18.932-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:18.932-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 201 checks in a row. 2016-11-02T14:13:19.299-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:19.299-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:19.434-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:19.434-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:19.800-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:19.800-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:19.935-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:19.935-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:20.304-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:20.304-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:20.439-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:20.439-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:20.806-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:20.806-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:20.944-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:20.944-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:21.311-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:21.311-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:21.448-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:21.448-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:21.448-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 211 checks in a row. 2016-11-02T14:13:21.813-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:21.813-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:21.954-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:21.954-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:22.317-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:22.317-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:22.460-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:22.460-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:22.818-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:22.818-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:22.961-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:22.961-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:23.321-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:23.321-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:23.461-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:23.461-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:23.824-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:23.825-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:23.964-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:23.964-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:23.964-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 221 checks in a row. 2016-11-02T14:13:24.327-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:24.327-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:24.465-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:24.465-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:24.830-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:24.830-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:24.966-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:24.967-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:25.335-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:25.335-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:25.469-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:25.469-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:25.839-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:25.840-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:25.840-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:13:25.974-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:25.974-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:26.479-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:26.480-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:26.985-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:26.985-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:26.985-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 231 checks in a row. 2016-11-02T14:13:27.487-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:27.487-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:27.991-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:27.991-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:28.492-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:28.492-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:28.993-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:28.993-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:29.497-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:29.497-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:30.000-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:30.000-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:30.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:30.506-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:31.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:31.010-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:31.515-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:31.515-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:32.016-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:32.016-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:32.016-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 241 checks in a row. 2016-11-02T14:13:32.518-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:32.518-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:33.022-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:33.022-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:33.526-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:33.526-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:34.027-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:34.027-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:34.529-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:34.529-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:35.031-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.031-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:35.515-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.515-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:35.533-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.533-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:35.533-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:13:35.544-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.544-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:35.844-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.844-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:35.844-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 251 checks in a row. 2016-11-02T14:13:36.021-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.021-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:36.044-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.044-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:36.348-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.348-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:36.524-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.524-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:36.545-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.545-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:36.849-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.849-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:37.027-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.028-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:37.049-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.049-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:37.354-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.354-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:37.532-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.532-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:37.532-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 261 checks in a row. 2016-11-02T14:13:37.550-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.550-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:37.859-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.859-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:38.037-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.037-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:38.053-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.053-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:38.362-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.362-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:38.539-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.539-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:38.554-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.554-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:38.863-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.863-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:39.040-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.040-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:39.056-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.057-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:39.057-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 271 checks in a row. 2016-11-02T14:13:39.364-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.365-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:39.545-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.545-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:39.559-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.559-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:39.870-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.870-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:40.051-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.051-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.062-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.062-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:40.373-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.373-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:40.556-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.556-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.568-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.568-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:40.876-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.876-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:40.876-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 281 checks in a row. 2016-11-02T14:13:41.061-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.061-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.069-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.069-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.246-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed asio heartbeat to :27017 - HostUnreachable: End of file 2016-11-02T14:13:41.246-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to close stream: Socket is not connected 2016-11-02T14:13:41.247-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed asio heartbeat to :27017 - HostUnreachable: End of file 2016-11-02T14:13:41.247-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to close stream: Socket is not connected 2016-11-02T14:13:41.377-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.377-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:41.562-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.562-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.574-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.574-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.878-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.878-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:42.067-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.067-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.077-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.077-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.382-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.382-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:42.572-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.572-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.572-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 291 checks in a row. 2016-11-02T14:13:42.583-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.583-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.883-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.883-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:43.076-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.076-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.085-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.085-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.385-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.385-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:43.579-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.579-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.587-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.587-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.887-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.887-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:44.082-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.082-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.090-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.090-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.090-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 301 checks in a row. 2016-11-02T14:13:44.389-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.389-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:44.587-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.587-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.593-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.593-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.890-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.890-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:45.093-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.093-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.099-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.099-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.394-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.394-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:45.399-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.399-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:45.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:45.596-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.596-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.596-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 311 checks in a row. 2016-11-02T14:13:45.601-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.601-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.899-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.900-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:45.904-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.904-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:46.101-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.101-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:46.102-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.102-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:46.402-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.402-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:46.405-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.405-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:46.606-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.606-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:46.906-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.906-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:46.908-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.908-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:46.908-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 321 checks in a row. 2016-11-02T14:13:47.110-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.110-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:47.410-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.410-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:47.615-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.615-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:47.911-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.911-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:48.120-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.120-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.416-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.416-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:48.624-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.624-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:49.129-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.129-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.419-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.419-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:49.419-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 331 checks in a row. 2016-11-02T14:13:49.630-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.630-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.922-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.922-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:50.134-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.134-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.423-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.423-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:50.640-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.640-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.929-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.929-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:51.146-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.146-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:51.432-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.432-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:51.647-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.647-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:51.934-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.934-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:51.934-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 341 checks in a row. 2016-11-02T14:13:52.148-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.148-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:52.436-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.437-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:52.652-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.653-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:52.942-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.942-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:53.154-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.154-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.443-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.443-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:53.660-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.660-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.946-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.946-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:54.163-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.163-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:54.448-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.448-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:54.448-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 351 checks in a row. 2016-11-02T14:13:54.668-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.668-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:54.950-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.950-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:55.172-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.172-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:55.452-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.452-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:55.673-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.673-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.673-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:13:55.673-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:13:55.953-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.953-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:55.953-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.953-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:56.456-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.456-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:56.957-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.957-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:57.458-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.458-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:57.458-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 361 checks in a row. 2016-11-02T14:13:57.959-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.959-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:13:58.463-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.463-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:58.969-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.969-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:59.475-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.475-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:13:59.978-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.978-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:00.481-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:00.481-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:00.985-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:00.985-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:01.486-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:01.486-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:01.987-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:01.987-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:02.490-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:02.490-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:02.490-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 371 checks in a row. 2016-11-02T14:14:02.994-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:02.994-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:03.497-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:03.497-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:04.001-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:04.002-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:04.505-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:04.505-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:05.008-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:05.008-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:05.509-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:05.509-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:05.509-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:14:06.011-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:06.011-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:06.511-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:06.512-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:07.016-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:07.016-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:07.516-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:07.517-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:07.517-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 381 checks in a row. 2016-11-02T14:14:08.017-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:08.017-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:08.518-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:08.518-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:09.020-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:09.020-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:09.521-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:09.521-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:10.025-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:10.025-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:10.530-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:10.530-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:11.036-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:11.036-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:11.540-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:11.540-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:12.042-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:12.043-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:12.545-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:12.546-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:12.546-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 391 checks in a row. 2016-11-02T14:14:13.047-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:13.047-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:13.550-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:13.550-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:14.051-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:14.051-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:14.552-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:14.552-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:15.056-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:15.056-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:15.400-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:15.400-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:15.557-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:15.557-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:15.903-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:15.903-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:16.058-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:16.058-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:16.058-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:14:16.405-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:16.405-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:16.405-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 401 checks in a row. 2016-11-02T14:14:16.906-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:16.906-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:17.408-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:17.408-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:17.909-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:17.909-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:18.410-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:18.410-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:18.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:18.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:19.415-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:19.415-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:19.918-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:19.918-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:20.424-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:20.424-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:20.928-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:20.929-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:21.432-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:21.432-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:21.432-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 411 checks in a row. 2016-11-02T14:14:21.936-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:21.936-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:22.438-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:22.438-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:22.939-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:22.939-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:23.440-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:23.440-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:23.941-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:23.942-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:24.446-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:24.446-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:24.952-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:24.952-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:25.454-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:25.454-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:25.676-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:25.676-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:25.959-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:25.959-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:25.959-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 421 checks in a row. 2016-11-02T14:14:26.062-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.063-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:26.179-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.179-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:26.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.464-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:26.567-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.567-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:26.684-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.684-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:26.966-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.966-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:27.071-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.071-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:27.187-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.187-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:27.467-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.467-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:27.573-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.573-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:27.573-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 431 checks in a row. 2016-11-02T14:14:27.689-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.689-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:27.969-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.969-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:28.075-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.075-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:28.192-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.192-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:28.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:28.577-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.577-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:28.695-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.695-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:28.974-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.974-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:29.078-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.078-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:29.200-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.200-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:29.200-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 441 checks in a row. 2016-11-02T14:14:29.480-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.480-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:29.580-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.580-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:29.704-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.704-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:29.985-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.985-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:30.083-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.083-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:30.207-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.207-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:30.486-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.486-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:30.585-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.585-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:30.711-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.711-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:30.988-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.988-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:30.988-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 451 checks in a row. 2016-11-02T14:14:31.086-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.086-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:31.215-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.215-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.489-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.490-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:31.589-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.590-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:31.718-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.718-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:32.094-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.094-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:32.224-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.224-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:32.498-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.499-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:32.594-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.595-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:32.595-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 461 checks in a row. 2016-11-02T14:14:32.729-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.729-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:33.002-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.002-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:33.097-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.097-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:33.234-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.234-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:33.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.507-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:33.598-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.598-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:33.740-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.740-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.008-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.008-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:34.099-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.099-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:34.241-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.241-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.241-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 471 checks in a row. 2016-11-02T14:14:34.510-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.510-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:34.604-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.604-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:34.743-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.743-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:35.013-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.013-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:35.109-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.109-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:35.248-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.248-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:35.516-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.516-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:35.516-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:14:35.615-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.615-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:35.749-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.750-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:36.120-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.120-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:36.121-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 481 checks in a row. 2016-11-02T14:14:36.254-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.254-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:36.622-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.622-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:36.755-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.755-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:37.125-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.125-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:37.261-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.261-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:37.626-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.626-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:37.766-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.766-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:38.127-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.127-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:38.270-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.271-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:38.633-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.633-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:38.633-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 491 checks in a row. 2016-11-02T14:14:38.772-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.772-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:39.135-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.135-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:39.274-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.274-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:39.641-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.641-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:39.776-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.776-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.142-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.142-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:40.279-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.279-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.644-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.644-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:40.782-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.782-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:41.145-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.145-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:41.145-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 501 checks in a row. 2016-11-02T14:14:41.283-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.283-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:41.649-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.650-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:41.789-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.789-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.151-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.151-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:42.293-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.293-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.656-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.656-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:42.794-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.794-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:43.159-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.159-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:43.299-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.299-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:43.662-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.662-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:43.662-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 511 checks in a row. 2016-11-02T14:14:43.800-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.800-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:44.164-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.164-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:44.302-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.302-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:44.669-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.669-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:44.803-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.803-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.170-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.170-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:45.309-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.309-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:14:45.671-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.671-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:45.815-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.815-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.815-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 521 checks in a row. 2016-11-02T14:14:45.815-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:14:45.815-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:14:45.902-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.902-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:46.172-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.172-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:46.172-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.173-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:46.406-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.406-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:46.674-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.674-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:46.907-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.907-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:47.181-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.181-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:47.409-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.409-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:47.686-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.686-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:47.914-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.914-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:47.914-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 531 checks in a row. 2016-11-02T14:14:48.190-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.190-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:48.415-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.415-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:48.691-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.691-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:48.919-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.919-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:49.193-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.193-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:49.423-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.423-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:49.695-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.695-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:49.928-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.928-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:50.201-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.201-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:50.431-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.431-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:50.431-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 541 checks in a row. 2016-11-02T14:14:50.705-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.705-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:50.934-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.934-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:51.207-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:51.207-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:51.436-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:51.436-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:51.709-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:51.709-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:51.939-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:51.939-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:52.210-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:52.210-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:52.442-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:52.442-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:52.714-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:52.714-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:52.944-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:52.944-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:52.944-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 551 checks in a row. 2016-11-02T14:14:53.215-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:53.215-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:53.447-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:53.447-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:53.718-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:53.718-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:53.950-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:53.950-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:54.221-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:54.221-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:54.453-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:54.453-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:54.726-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:54.726-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:54.955-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:54.955-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:55.227-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:55.227-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:55.459-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:55.459-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:55.459-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 561 checks in a row. 2016-11-02T14:14:55.728-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:55.728-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:55.962-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:55.962-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:56.233-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:56.233-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:56.467-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:56.467-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:56.735-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:56.735-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:56.969-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:56.969-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:57.236-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:57.236-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:57.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:57.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:57.737-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:57.737-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:57.979-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:57.980-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:57.980-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 571 checks in a row. 2016-11-02T14:14:58.242-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:58.242-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:58.483-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:58.483-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:58.744-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:58.745-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:58.985-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:58.985-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:59.245-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:59.245-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:59.490-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:59.490-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:14:59.747-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:59.747-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:14:59.991-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:59.991-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:00.252-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:00.252-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:00.492-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:00.492-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:00.492-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 581 checks in a row. 2016-11-02T14:15:00.757-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:00.757-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:00.996-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:00.996-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:01.262-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:01.262-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:01.497-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:01.497-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:01.764-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:01.764-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:01.999-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:01.999-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:02.267-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:02.267-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:02.501-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:02.501-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:02.769-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:02.769-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:03.002-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:03.002-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:03.002-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 591 checks in a row. 2016-11-02T14:15:03.270-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:03.270-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:03.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:03.506-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:03.774-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:03.775-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:04.007-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:04.007-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:04.276-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:04.276-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:04.507-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:04.508-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:04.779-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:04.779-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:05.012-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:05.012-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:05.281-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:05.281-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:05.515-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:05.515-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:05.515-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 601 checks in a row. 2016-11-02T14:15:05.515-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:15:05.785-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:05.785-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:06.288-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:06.288-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:06.288-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:15:15.400-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.400-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:15.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:15:15.815-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.815-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:15.902-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.902-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:16.292-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.292-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:16.320-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.320-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:16.406-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.406-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:16.793-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.793-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:16.793-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 611 checks in a row. 2016-11-02T14:15:16.824-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.825-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:16.908-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.908-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:17.297-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.297-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:17.327-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.327-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:17.411-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.411-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:17.800-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.800-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:17.828-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.828-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:17.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:18.300-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.301-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:18.329-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.329-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:18.329-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 621 checks in a row. 2016-11-02T14:15:18.414-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.414-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:18.806-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.806-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:18.830-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.831-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:18.914-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.914-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:19.307-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.307-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:19.335-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.335-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:19.415-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.415-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:19.808-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.808-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:19.837-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.837-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:19.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:19.917-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 631 checks in a row. 2016-11-02T14:15:20.310-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.310-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:20.340-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.340-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:20.421-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.421-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:20.814-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.814-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:20.845-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.845-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:20.926-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.926-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:21.315-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.315-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:21.347-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.347-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:21.429-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.429-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:21.818-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.818-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:21.818-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 641 checks in a row. 2016-11-02T14:15:21.852-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.852-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:21.930-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.930-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:22.322-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.322-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:22.356-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.357-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:22.431-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.431-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:22.826-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.826-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:22.862-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.862-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:22.933-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.933-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:23.327-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.327-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:23.365-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.365-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.365-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 651 checks in a row. 2016-11-02T14:15:23.437-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.437-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:23.832-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.832-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:23.868-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.868-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.941-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.941-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:24.334-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.334-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:24.373-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.373-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.444-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.444-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:24.836-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.836-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:24.875-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.875-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.945-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.945-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:24.945-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 661 checks in a row. 2016-11-02T14:15:25.341-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.341-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:25.378-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.378-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:25.446-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.446-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:25.842-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.842-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:25.882-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.882-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:25.947-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.947-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:26.343-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.343-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:26.383-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.383-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:26.452-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.452-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:26.846-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.846-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:26.846-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 671 checks in a row. 2016-11-02T14:15:26.885-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.885-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:26.953-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.953-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:27.347-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.347-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:27.390-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.390-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:27.456-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.456-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:27.849-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.849-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:27.893-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.893-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:27.957-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.957-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:28.353-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.353-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:28.394-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.394-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:28.394-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 681 checks in a row. 2016-11-02T14:15:28.459-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.459-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:28.856-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.856-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:28.895-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.896-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:28.962-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.962-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:29.360-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.360-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:29.401-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.401-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:29.466-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.466-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:29.864-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.865-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:29.902-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.902-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:29.968-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.968-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:29.968-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 691 checks in a row. 2016-11-02T14:15:30.365-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.365-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:30.403-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.403-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:30.470-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.470-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:30.867-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.867-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:30.904-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.904-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:30.973-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.973-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:31.370-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.370-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:31.406-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.406-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:31.872-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.873-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:31.873-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 701 checks in a row. 2016-11-02T14:15:31.910-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.910-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.977-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.977-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:32.374-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.374-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:32.414-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.414-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:32.483-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.483-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:32.878-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.878-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:32.917-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.917-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:32.984-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.984-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:33.381-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.381-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:33.421-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.421-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:33.421-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 711 checks in a row. 2016-11-02T14:15:33.488-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.488-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:33.885-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.885-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:33.922-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.922-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:33.989-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.989-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:34.388-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.388-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:34.423-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.423-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:34.491-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.491-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:34.892-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.892-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:34.925-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.925-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:34.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:34.992-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 721 checks in a row. 2016-11-02T14:15:35.396-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.396-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:35.429-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.429-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:35.497-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.497-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:35.497-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:15:35.898-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.898-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:35.932-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.932-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:35.932-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:15:35.932-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:15:36.403-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.403-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:36.403-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.403-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:36.905-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.905-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:37.408-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.408-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:37.910-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.910-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:37.911-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 731 checks in a row. 2016-11-02T14:15:38.414-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.414-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:38.917-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.917-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:39.422-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.422-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:39.923-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.923-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:40.428-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.428-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:40.930-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.930-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:41.435-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:41.435-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:41.937-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:41.937-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:42.441-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:42.442-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:42.945-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:42.945-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:42.945-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 741 checks in a row. 2016-11-02T14:15:43.448-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:43.448-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:43.952-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:43.953-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:44.457-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:44.457-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:44.958-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:44.958-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:45.400-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.400-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:45.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.401-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:15:45.460-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.460-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:45.906-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.906-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:45.962-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.962-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:46.408-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:46.408-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:46.408-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 751 checks in a row. 2016-11-02T14:15:46.465-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:46.466-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:46.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:46.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:46.970-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:46.970-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:47.417-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:47.417-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:47.473-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:47.473-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:47.918-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:47.919-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:47.978-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:47.978-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:48.420-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:48.420-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:48.479-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:48.479-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:48.921-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:48.921-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:48.921-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 761 checks in a row. 2016-11-02T14:15:48.982-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:48.982-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:49.425-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:49.425-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:49.485-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:49.485-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:49.931-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:49.931-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:49.987-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:49.987-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:50.432-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:50.433-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:50.491-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:50.491-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:50.938-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:50.938-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:50.994-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:50.994-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:51.442-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:51.442-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:51.442-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 771 checks in a row. 2016-11-02T14:15:51.499-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:51.499-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:51.946-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:51.946-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:52.000-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:52.000-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:52.447-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:52.447-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:52.504-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:52.504-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:52.951-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:52.951-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:53.009-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:53.009-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:53.453-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:53.453-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:53.512-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:53.512-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:53.955-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:53.955-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:53.955-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 781 checks in a row. 2016-11-02T14:15:54.013-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:54.013-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:54.456-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:54.456-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:54.516-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:54.516-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:54.957-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:54.957-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:55.017-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:55.017-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:55.461-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:55.461-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:55.522-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:55.522-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:55.966-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:55.966-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:56.024-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:56.024-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:56.471-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:56.471-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:56.471-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 791 checks in a row. 2016-11-02T14:15:56.528-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:56.528-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:15:56.528-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:15:56.973-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:56.973-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:57.477-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:57.477-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:57.983-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:57.983-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:58.484-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:58.484-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:58.989-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:58.989-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:59.492-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:59.492-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:15:59.995-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:59.995-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:00.500-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:00.500-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:01.004-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:01.004-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:01.004-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 801 checks in a row. 2016-11-02T14:16:01.509-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:01.509-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:02.013-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:02.013-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:02.514-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:02.514-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:03.016-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:03.016-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:03.518-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:03.518-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:04.021-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:04.021-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:04.522-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:04.523-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:05.027-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:05.027-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:05.530-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:05.530-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:05.530-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:16:05.931-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:05.931-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:05.931-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 811 checks in a row. 2016-11-02T14:16:05.932-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:05.932-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:06.436-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:06.436-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:06.532-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:06.532-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:06.937-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:06.937-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:07.033-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.033-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:07.442-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.443-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:07.536-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.536-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:07.946-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.946-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:08.038-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.038-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:08.448-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.448-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:08.448-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 821 checks in a row. 2016-11-02T14:16:08.542-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.542-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:08.950-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.950-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:09.046-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.046-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:09.454-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.454-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:09.550-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.550-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:09.959-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.959-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:10.052-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.052-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:10.461-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.461-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:10.558-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.558-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:10.964-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.964-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:10.964-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 831 checks in a row. 2016-11-02T14:16:11.061-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.061-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:11.466-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.466-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:11.565-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.565-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:11.970-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.970-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:12.066-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.066-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:12.471-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.471-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:12.571-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.571-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:12.975-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.975-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:13.075-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.075-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:13.480-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.480-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:13.480-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 841 checks in a row. 2016-11-02T14:16:13.576-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.576-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:13.984-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.984-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:14.078-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.078-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:14.488-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.488-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:14.581-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.581-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:14.989-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.989-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:15.082-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.082-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:15.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:15.495-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.495-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:15.583-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.583-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:15.583-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 851 checks in a row. 2016-11-02T14:16:15.904-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.904-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:15.999-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.999-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:16.088-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.088-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:16.406-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.406-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:16.504-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.504-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:16.591-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.591-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:16.908-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.908-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:17.006-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.006-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:17.093-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.093-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:17.409-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.410-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:17.410-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 861 checks in a row. 2016-11-02T14:16:17.510-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.511-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:17.598-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.598-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:17.915-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.915-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:18.016-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.016-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.099-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.099-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:18.416-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.416-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:18.522-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.522-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.600-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.600-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:18.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:19.025-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.025-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:19.025-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 871 checks in a row. 2016-11-02T14:16:19.101-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.101-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:19.419-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.419-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:19.528-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.528-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:19.605-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.605-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:19.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.924-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:20.030-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.030-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:20.107-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.107-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:20.425-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.425-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:20.533-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.533-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:20.609-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.609-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:20.609-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 881 checks in a row. 2016-11-02T14:16:20.926-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.926-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:21.035-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.035-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.111-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.111-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:21.431-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.431-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:21.540-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.540-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.613-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.613-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:21.935-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.935-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:22.044-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.044-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:22.119-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.119-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:22.438-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.438-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:22.438-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 891 checks in a row. 2016-11-02T14:16:22.546-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.547-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:22.625-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.625-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:22.943-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.943-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:23.047-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.048-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.126-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.126-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:23.448-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.448-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:23.552-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.552-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:23.630-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.630-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:23.952-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.952-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:24.057-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.057-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:24.057-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 901 checks in a row. 2016-11-02T14:16:24.132-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.132-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:24.454-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.454-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:24.558-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.558-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:24.632-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.632-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:24.955-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.955-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:25.059-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.059-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:25.136-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.136-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:25.456-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.456-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:25.560-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.560-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:25.638-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.638-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:25.638-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 911 checks in a row. 2016-11-02T14:16:25.958-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.959-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:26.063-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.063-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:26.063-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:16:26.063-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:16:26.141-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.141-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:26.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.464-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:26.643-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.643-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:26.644-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.644-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:26.970-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.970-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:27.144-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.144-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:27.472-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.472-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:27.646-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.646-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:27.646-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 921 checks in a row. 2016-11-02T14:16:27.973-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.973-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:28.152-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.152-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:28.477-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.477-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:28.656-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.656-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:28.977-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.978-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:29.158-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.159-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:29.479-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.479-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:29.659-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.659-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:29.984-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.984-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:30.162-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.162-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:30.162-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 931 checks in a row. 2016-11-02T14:16:30.485-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.485-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:30.668-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.668-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:30.986-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.986-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:31.170-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:31.170-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:31.491-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:31.491-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:31.671-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:31.671-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:31.993-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:31.993-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:32.177-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:32.178-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:32.498-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:32.498-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:32.682-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:32.682-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:32.682-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 941 checks in a row. 2016-11-02T14:16:33.001-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:33.001-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:33.183-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:33.183-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:33.502-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:33.502-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:33.686-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:33.686-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:34.004-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:34.004-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:34.191-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:34.191-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:34.505-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:34.505-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:34.692-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:34.692-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:35.011-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:35.011-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:35.197-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:35.197-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:35.197-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 951 checks in a row. 2016-11-02T14:16:35.512-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:35.512-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:35.512-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:16:35.701-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:35.701-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:36.202-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:36.202-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:36.705-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:36.705-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:37.206-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:37.206-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:37.707-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:37.707-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:38.211-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:38.211-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:38.712-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:38.712-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:39.212-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:39.213-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:39.713-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:39.713-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:39.713-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 961 checks in a row. 2016-11-02T14:16:40.217-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:40.217-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:40.719-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:40.719-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:41.224-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:41.224-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:41.730-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:41.730-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:42.233-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:42.233-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:42.734-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:42.734-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:43.238-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:43.238-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:43.743-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:43.743-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:44.247-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:44.247-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:44.748-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:44.748-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:44.748-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 971 checks in a row. 2016-11-02T14:16:45.250-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.250-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:45.403-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.403-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:45.756-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.756-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:45.904-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.904-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:46.259-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:46.259-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:46.407-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:46.407-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:46.760-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:46.760-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:46.760-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:16:46.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:46.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:47.416-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:47.416-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:47.416-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 981 checks in a row. 2016-11-02T14:16:47.919-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:47.919-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:48.423-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:48.423-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:48.925-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:48.925-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:49.429-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:49.429-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:49.932-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:49.932-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:50.436-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:50.436-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:50.937-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:50.938-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:51.440-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:51.440-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:51.941-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:51.941-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:52.445-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:52.445-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:52.445-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 991 checks in a row. 2016-11-02T14:16:52.949-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:52.949-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:53.452-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:53.452-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:53.956-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:53.956-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:54.457-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:54.458-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:54.961-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:54.961-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:55.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:55.464-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:55.965-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:55.965-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:56.063-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.063-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:56.468-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.468-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:56.567-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.567-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:56.567-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1001 checks in a row. 2016-11-02T14:16:56.762-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.762-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:56.971-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.971-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:57.071-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.071-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:57.264-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.264-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:57.473-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.473-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:57.575-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.575-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:57.764-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.764-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:57.977-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.977-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:58.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:58.270-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.270-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:58.270-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1011 checks in a row. 2016-11-02T14:16:58.481-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.481-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:58.581-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.581-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:58.773-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.773-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:58.985-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.985-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:59.086-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.086-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:59.274-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.274-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:59.488-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.488-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:59.588-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.588-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:59.780-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.780-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:16:59.988-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.988-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:16:59.988-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1021 checks in a row. 2016-11-02T14:17:00.094-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.094-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:00.281-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.282-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:00.490-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.490-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:00.597-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.597-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:00.783-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.783-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:00.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:01.098-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.098-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:01.288-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.288-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:01.496-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.496-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:01.602-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.602-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:01.602-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1031 checks in a row. 2016-11-02T14:17:01.788-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.788-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:01.998-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.998-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:02.103-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.103-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:02.289-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.289-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:02.500-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.501-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:02.608-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.608-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:02.790-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.790-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:03.004-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.004-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:03.114-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.114-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:03.292-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.292-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:03.292-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1041 checks in a row. 2016-11-02T14:17:03.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.506-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:03.620-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.620-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:03.794-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.794-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:04.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.010-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:04.123-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.123-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:04.296-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.296-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:04.514-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.514-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:04.624-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.624-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:04.797-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.797-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:05.018-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.018-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:05.018-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1051 checks in a row. 2016-11-02T14:17:05.127-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.127-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:05.299-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.299-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:05.521-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.521-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:05.521-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:17:05.629-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.630-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:05.805-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.805-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:06.134-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.134-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.307-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.307-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:06.636-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.636-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.811-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.811-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:07.137-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.137-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:07.137-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1061 checks in a row. 2016-11-02T14:17:07.314-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.314-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:07.641-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.641-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:07.815-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.815-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:08.144-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.144-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:08.317-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.317-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:08.647-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.648-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:08.820-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.820-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:09.151-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.151-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:09.326-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.326-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:09.655-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.655-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:09.655-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1071 checks in a row. 2016-11-02T14:17:09.829-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.829-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:10.156-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.156-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:10.330-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.330-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:10.658-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.658-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:10.831-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.832-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:11.162-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.162-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.334-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.334-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:11.666-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.666-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.839-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.839-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:12.168-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.168-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:12.168-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1081 checks in a row. 2016-11-02T14:17:12.344-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.345-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:12.670-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.670-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:12.850-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.850-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:13.174-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.174-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.351-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.351-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:13.675-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.675-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.854-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.854-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:14.177-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.178-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.355-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.355-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:14.683-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.683-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.683-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1091 checks in a row. 2016-11-02T14:17:14.857-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.857-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:15.187-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.187-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.359-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.359-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:15.403-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.403-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:15.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:15.691-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.691-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.861-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.861-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:15.907-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.907-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:16.196-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.196-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:16.196-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:17:16.197-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:17:16.365-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.365-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:16.365-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1101 checks in a row. 2016-11-02T14:17:16.408-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.408-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:16.869-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.869-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:16.869-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.869-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:16.910-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.910-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:17.370-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.370-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:17.410-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.412-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:17.875-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.875-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:17.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:18.377-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.377-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:18.418-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.418-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:18.418-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1111 checks in a row. 2016-11-02T14:17:18.880-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.880-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:18.922-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.922-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:19.380-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.381-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:19.423-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.423-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:19.882-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.883-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:19.925-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.925-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:20.386-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.386-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:20.428-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.428-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:20.891-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.891-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:20.929-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.929-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:20.929-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1121 checks in a row. 2016-11-02T14:17:21.396-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:21.396-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:21.432-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:21.432-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:21.897-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:21.897-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:21.934-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:21.934-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:22.399-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:22.400-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:22.440-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:22.440-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:22.902-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:22.902-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:22.942-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:22.943-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:23.404-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:23.405-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:23.448-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:23.448-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:23.448-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1131 checks in a row. 2016-11-02T14:17:23.908-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:23.908-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:23.949-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:23.949-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:24.409-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:24.409-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:24.452-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:24.452-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:24.913-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:24.913-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:24.954-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:24.954-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:25.414-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:25.414-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:25.459-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:25.459-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:25.918-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:25.919-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:25.962-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:25.962-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:25.962-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1141 checks in a row. 2016-11-02T14:17:26.420-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:26.421-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:26.465-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:26.465-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:26.926-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:26.926-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:26.968-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:26.969-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:27.428-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:27.428-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:27.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:27.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:27.934-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:27.934-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:27.978-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:27.978-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:28.438-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:28.438-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:28.480-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:28.480-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:28.480-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1151 checks in a row. 2016-11-02T14:17:28.942-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:28.942-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:28.981-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:28.981-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:29.443-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:29.443-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:29.482-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:29.482-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:29.944-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:29.945-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:29.983-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:29.983-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:30.445-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:30.445-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:30.486-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:30.486-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:30.948-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:30.949-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:30.989-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:30.989-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:30.989-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1161 checks in a row. 2016-11-02T14:17:31.453-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:31.453-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:31.495-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:31.495-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:31.955-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:31.955-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:32.000-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:32.000-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:32.460-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:32.460-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:32.504-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:32.504-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:32.965-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:32.965-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:33.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:33.010-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:33.470-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:33.470-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:33.513-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:33.513-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:33.513-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1171 checks in a row. 2016-11-02T14:17:33.974-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:33.974-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:34.016-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:34.016-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:34.476-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:34.476-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:34.521-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:34.521-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:34.980-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:34.980-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:35.026-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:35.027-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:35.485-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:35.485-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:35.531-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:35.531-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:35.531-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:17:35.988-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:35.988-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:36.490-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:36.490-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:36.491-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1181 checks in a row. 2016-11-02T14:17:36.992-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:36.992-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:36.992-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:17:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:45.907-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:45.907-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:46.200-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.200-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:46.410-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.410-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:46.706-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.706-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:46.911-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.911-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:46.997-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.997-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:47.209-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.209-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:47.412-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.412-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:47.412-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1191 checks in a row. 2016-11-02T14:17:47.498-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.498-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:47.712-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.712-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:47.915-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.915-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:48.000-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.000-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:48.216-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.216-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:48.415-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.415-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:48.501-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.501-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:48.721-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.722-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:48.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:49.005-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.005-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:49.005-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1201 checks in a row. 2016-11-02T14:17:49.224-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.224-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:49.421-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.421-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:49.510-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.510-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:49.728-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.728-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:49.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.925-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:50.015-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.015-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:50.234-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.234-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:50.425-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.425-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:50.516-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.516-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:50.736-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.736-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:50.736-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1211 checks in a row. 2016-11-02T14:17:50.928-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.928-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:51.021-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.021-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:51.241-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.241-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:51.429-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.429-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:51.524-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.524-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:51.743-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.743-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:51.930-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.930-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:52.028-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.028-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:52.246-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.246-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.430-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.430-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:52.430-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1221 checks in a row. 2016-11-02T14:17:52.529-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.529-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:52.747-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.747-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.931-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.931-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:53.031-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.032-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:53.248-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.248-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.433-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.433-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:53.532-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.532-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:53.750-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.750-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.936-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.936-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:54.037-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.037-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:54.037-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1231 checks in a row. 2016-11-02T14:17:54.253-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.253-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:54.437-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.437-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:54.543-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.543-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:54.756-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.756-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:54.940-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.940-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:55.047-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.048-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:55.259-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.259-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:55.443-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.443-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:55.552-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.552-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:55.765-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.765-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:55.765-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1241 checks in a row. 2016-11-02T14:17:55.944-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.944-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:56.052-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.053-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:56.268-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.268-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:56.447-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.447-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:56.553-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.553-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:56.770-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.770-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:56.948-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.948-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:57.054-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.054-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:57.276-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.276-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:57.451-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.451-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:57.451-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1251 checks in a row. 2016-11-02T14:17:57.560-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.560-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:57.777-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.777-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:57.953-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.953-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:58.061-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.061-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:58.281-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.281-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:58.459-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.459-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:58.563-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.563-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:58.782-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.782-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:58.963-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.964-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:59.064-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.064-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:59.064-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1261 checks in a row. 2016-11-02T14:17:59.286-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.286-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.468-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.468-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:17:59.567-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.567-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:17:59.787-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.787-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.973-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.973-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:00.068-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.068-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:00.293-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.293-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:00.477-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.477-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:00.569-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.569-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:00.796-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.796-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:00.796-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1271 checks in a row. 2016-11-02T14:18:00.978-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.978-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:01.070-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.070-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:01.298-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.298-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:01.479-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.479-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:01.576-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.576-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:01.800-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.800-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:01.983-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.983-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:02.078-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.078-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:02.306-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.306-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:02.487-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.487-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:02.487-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1281 checks in a row. 2016-11-02T14:18:02.579-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.579-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:02.811-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.811-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:02.991-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.991-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:03.080-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.080-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:03.314-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.314-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.497-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.497-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:03.582-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.582-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:03.820-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.820-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.999-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.999-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:04.085-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.085-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:04.085-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1291 checks in a row. 2016-11-02T14:18:04.321-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.321-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:04.503-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.503-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:04.586-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.587-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:04.823-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.823-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:05.007-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.007-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:05.087-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.087-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:05.324-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.324-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:05.510-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.510-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:05.510-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:18:05.591-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.591-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:05.830-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.830-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:05.830-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1301 checks in a row. 2016-11-02T14:18:06.092-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.092-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:06.336-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.337-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:06.337-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:18:06.337-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:18:06.595-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.595-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:07.100-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.100-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:07.100-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.100-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:07.601-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.601-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:08.102-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.102-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:08.604-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.604-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:09.107-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.107-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:09.610-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.610-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:09.610-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1311 checks in a row. 2016-11-02T14:18:10.113-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.113-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:10.617-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.617-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:11.119-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:11.119-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:11.623-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:11.623-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:12.125-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:12.125-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:12.631-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:12.631-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:13.132-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:13.132-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:13.633-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:13.633-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:14.135-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:14.135-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:14.640-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:14.640-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:14.640-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1321 checks in a row. 2016-11-02T14:18:15.143-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:15.144-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:15.406-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:15.406-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:15.645-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:15.645-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:15.908-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:15.908-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:16.148-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:16.148-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:16.413-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:16.413-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:16.654-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:16.654-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:16.916-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:16.916-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:17.155-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:17.155-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:17.418-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:17.418-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:17.418-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1331 checks in a row. 2016-11-02T14:18:17.661-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:17.661-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:17.921-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:17.921-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:18.162-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:18.162-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:18.424-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:18.424-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:18.665-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:18.665-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:18.929-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:18.929-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:19.166-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:19.166-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:19.430-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:19.430-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:19.667-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:19.667-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:19.931-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:19.931-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:19.931-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1341 checks in a row. 2016-11-02T14:18:20.171-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:20.171-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:20.436-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:20.436-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:20.673-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:20.673-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:20.936-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:20.937-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:21.178-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:21.178-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:21.439-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:21.439-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:21.678-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:21.679-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:21.943-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:21.943-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:22.182-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:22.182-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:22.446-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:22.446-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:22.446-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1351 checks in a row. 2016-11-02T14:18:22.685-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:22.685-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:22.948-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:22.948-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:23.189-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:23.189-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:23.450-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:23.450-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:23.691-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:23.691-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:23.952-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:23.952-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:24.194-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:24.194-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:24.458-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:24.458-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:24.697-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:24.697-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:24.959-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:24.959-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:24.959-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1361 checks in a row. 2016-11-02T14:18:25.201-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:25.201-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:25.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:25.464-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:25.704-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:25.705-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:25.965-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:25.965-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:26.207-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:26.207-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:26.466-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:26.466-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:26.710-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:26.710-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:26.972-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:26.972-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:27.212-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:27.213-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:27.213-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:18:27.478-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:27.478-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:27.478-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1371 checks in a row. 2016-11-02T14:18:27.978-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:27.978-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:28.484-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:28.484-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:28.989-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:28.989-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:29.494-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:29.495-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:29.999-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:29.999-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:30.503-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:30.503-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:31.004-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:31.004-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:31.507-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:31.507-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:32.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:32.011-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:32.513-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:32.513-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:32.514-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1381 checks in a row. 2016-11-02T14:18:33.015-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:33.015-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:33.517-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:33.517-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:34.021-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:34.021-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:34.524-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:34.524-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:35.026-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:35.026-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:35.531-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:35.531-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:35.531-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:18:36.337-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:36.337-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:36.838-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:36.838-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:37.213-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.213-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:37.341-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.341-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:37.341-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1391 checks in a row. 2016-11-02T14:18:37.714-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.715-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:37.846-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.846-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:38.215-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.215-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:38.351-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.351-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:38.716-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.717-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:38.854-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.854-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:39.218-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.218-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:39.355-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.355-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:39.718-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.719-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:39.860-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.860-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:39.860-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1401 checks in a row. 2016-11-02T14:18:40.222-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.222-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:40.364-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.364-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:40.723-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.723-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:40.870-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.871-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.225-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.225-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:41.372-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.372-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.730-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.730-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:41.877-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.877-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:42.236-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.236-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:42.383-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.383-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:42.383-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1411 checks in a row. 2016-11-02T14:18:42.738-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.738-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:42.884-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.885-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:43.243-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.243-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:43.388-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.388-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:43.747-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.748-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:43.889-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.889-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:44.252-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.252-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:44.392-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.392-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:44.753-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.753-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:44.896-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.896-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:44.896-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1421 checks in a row. 2016-11-02T14:18:45.254-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.254-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:45.398-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.398-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:45.406-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.406-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:45.408-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.408-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:45.760-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.760-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:45.902-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.902-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:45.910-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.910-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:46.262-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.262-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:46.408-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.408-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.412-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.412-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:46.412-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1431 checks in a row. 2016-11-02T14:18:46.764-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.764-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:46.910-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.910-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:47.265-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.265-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:47.414-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.415-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:47.420-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.420-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:47.766-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.766-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:47.916-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.916-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:47.920-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.920-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:48.270-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.270-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:48.270-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1441 checks in a row. 2016-11-02T14:18:48.420-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.420-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:48.423-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.423-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:48.774-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.774-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:48.921-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.921-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:48.925-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.925-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:49.275-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.275-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:49.422-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.422-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:49.424-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.424-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:49.430-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.430-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:49.776-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.776-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:49.776-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1451 checks in a row. 2016-11-02T14:18:49.926-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.926-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:49.933-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.933-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:50.278-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.278-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:50.430-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.430-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:50.438-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.438-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:50.782-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.782-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:50.931-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.931-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:50.942-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.942-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:51.286-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.286-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:51.432-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.432-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:51.432-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1461 checks in a row. 2016-11-02T14:18:51.443-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.443-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:51.788-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.788-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:51.933-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.933-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:51.949-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.949-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:52.292-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.292-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:52.435-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.435-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:52.450-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.450-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:52.797-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.797-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:52.939-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.939-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:52.953-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.953-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:52.953-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1471 checks in a row. 2016-11-02T14:18:53.303-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.303-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:53.440-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.440-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.459-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.459-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:53.803-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.803-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:53.941-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.941-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.962-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.962-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:54.307-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.307-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:54.445-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.445-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:54.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.464-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:54.812-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.812-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:54.812-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1481 checks in a row. 2016-11-02T14:18:54.950-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.950-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:54.964-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.964-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:55.318-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.318-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:55.454-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.454-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.465-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.465-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:55.820-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.820-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:55.958-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.958-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.966-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.966-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:56.326-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.326-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:56.462-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.462-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:56.462-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1491 checks in a row. 2016-11-02T14:18:56.462-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:18:56.462-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:18:56.469-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.469-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:56.831-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.831-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:56.971-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.971-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:57.337-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.337-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:57.338-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.338-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:57.472-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.472-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:57.842-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.842-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:57.973-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.973-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:58.344-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.344-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:58.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:58.474-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1501 checks in a row. 2016-11-02T14:18:58.845-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.845-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:58.976-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.976-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:59.350-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.350-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:59.478-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.478-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:18:59.855-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.855-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:18:59.981-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.981-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:00.356-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.356-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:00.483-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.483-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:00.858-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.858-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:00.987-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.987-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:00.987-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1511 checks in a row. 2016-11-02T14:19:01.360-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:01.360-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:01.489-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:01.489-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:01.861-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:01.862-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:01.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:01.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:02.366-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:02.366-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:02.493-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:02.493-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:02.868-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:02.868-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:02.995-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:02.995-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:03.370-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:03.370-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:03.496-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:03.496-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:03.496-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1521 checks in a row. 2016-11-02T14:19:03.871-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:03.871-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:03.997-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:03.997-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:04.373-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:04.373-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:04.500-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:04.500-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:04.875-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:04.875-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:05.001-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:05.001-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:05.379-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:05.379-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:05.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:05.506-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:05.506-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:19:05.880-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:05.880-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:06.382-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:06.382-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:06.382-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1531 checks in a row. 2016-11-02T14:19:06.888-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:06.888-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:07.393-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:07.393-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:07.894-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:07.895-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:08.399-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:08.399-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:08.900-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:08.900-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:09.401-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:09.401-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:09.907-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:09.907-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:10.410-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:10.410-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:10.913-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:10.913-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:11.414-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:11.414-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:11.414-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1541 checks in a row. 2016-11-02T14:19:11.915-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:11.915-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:12.421-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:12.421-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:12.925-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:12.925-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:13.430-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:13.430-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:13.935-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:13.935-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:14.438-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:14.438-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:14.941-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:14.941-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:15.408-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:15.408-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:15.445-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:15.446-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:15.914-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:15.914-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:15.914-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1551 checks in a row. 2016-11-02T14:19:15.948-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:15.948-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:16.419-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:16.419-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:16.452-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:16.452-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:16.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:16.924-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:16.957-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:16.957-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:17.427-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:17.427-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:17.461-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:17.461-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:17.461-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:19:17.929-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:17.929-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:18.434-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:18.434-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:18.939-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:18.939-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:18.939-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1561 checks in a row. 2016-11-02T14:19:19.441-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:19.442-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:19.943-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:19.943-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:20.446-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:20.446-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:20.950-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:20.950-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:21.453-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:21.453-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:21.958-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:21.958-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:22.461-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:22.461-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:22.964-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:22.965-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:23.468-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:23.468-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:23.972-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:23.972-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:23.972-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1571 checks in a row. 2016-11-02T14:19:24.475-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:24.475-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:24.980-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:24.980-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:25.485-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:25.485-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:25.986-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:25.986-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:26.462-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.462-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:26.488-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.488-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:26.963-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.963-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:26.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:27.464-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.464-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:27.469-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.469-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:27.469-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1581 checks in a row. 2016-11-02T14:19:27.495-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.495-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:27.969-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.969-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:27.998-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.998-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:28.473-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:28.473-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:28.504-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:28.504-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:28.974-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:28.974-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:29.007-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.007-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:29.479-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.479-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:29.510-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.510-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:29.982-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.982-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:29.982-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1591 checks in a row. 2016-11-02T14:19:30.011-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.011-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:30.483-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.483-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:30.516-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.516-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:30.984-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.984-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:31.020-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.020-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:31.490-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.491-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:31.525-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.525-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:31.996-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.997-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:32.028-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.028-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:32.499-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.499-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:32.499-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1601 checks in a row. 2016-11-02T14:19:32.531-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.531-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:33.002-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.003-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:33.035-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.035-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:33.505-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.506-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:33.540-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.540-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:34.008-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.008-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:34.044-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.044-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:34.512-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.512-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:34.548-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.548-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:35.013-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.014-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:35.014-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1611 checks in a row. 2016-11-02T14:19:35.052-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.052-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:35.518-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.518-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:35.556-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.556-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:35.556-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:19:36.020-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.020-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:36.522-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.522-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:37.024-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.024-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:37.530-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.530-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:38.031-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.031-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:38.534-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.534-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:39.036-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.036-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:39.036-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1621 checks in a row. 2016-11-02T14:19:39.537-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.537-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:40.041-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.041-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:40.543-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.543-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.044-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.044-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.548-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.548-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:42.052-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.052-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:42.558-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.558-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:43.063-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.063-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:43.564-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.564-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:44.070-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.070-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:44.070-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1631 checks in a row. 2016-11-02T14:19:44.571-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.572-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:45.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:45.409-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.409-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:45.580-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.580-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:45.912-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.912-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:46.083-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.084-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:46.415-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.415-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:46.586-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.586-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:46.586-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:19:46.586-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:19:46.917-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.917-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:47.087-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.087-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:47.087-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1641 checks in a row. 2016-11-02T14:19:47.420-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.420-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:47.590-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.590-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:47.590-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.590-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:47.924-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.924-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:48.092-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.092-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:48.424-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.424-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:48.594-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.594-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:48.928-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.928-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:49.095-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.096-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:49.432-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.432-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:49.432-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1651 checks in a row. 2016-11-02T14:19:49.598-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.598-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:49.935-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.935-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:50.102-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.102-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:50.439-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.439-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:50.603-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.603-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:50.941-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.941-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:51.109-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:51.109-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:51.447-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:51.447-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:51.610-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:51.610-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:51.949-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:51.949-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:51.949-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1661 checks in a row. 2016-11-02T14:19:52.115-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:52.115-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:52.451-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:52.451-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:52.617-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:52.617-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:52.953-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:52.953-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:53.123-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:53.123-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:53.457-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:53.457-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:53.626-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:53.626-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:53.960-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:53.960-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:54.129-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:54.129-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:54.465-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:54.465-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:54.465-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1671 checks in a row. 2016-11-02T14:19:54.630-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:54.630-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:54.966-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:54.966-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:55.130-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:55.130-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:55.469-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:55.469-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:55.632-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:55.632-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:55.971-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:55.971-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:56.135-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:56.135-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:56.472-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:56.472-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:56.641-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:56.641-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:56.974-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:56.974-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:56.974-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1681 checks in a row. 2016-11-02T14:19:57.142-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:57.142-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:57.479-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:57.479-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:57.645-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:57.645-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:57.982-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:57.982-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:58.147-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:58.147-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:58.485-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:58.486-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:58.650-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:58.650-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:58.990-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:58.990-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:59.151-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:59.151-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:59.493-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:59.493-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:19:59.493-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1691 checks in a row. 2016-11-02T14:19:59.652-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:59.652-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:19:59.996-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:59.996-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:00.156-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:00.156-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:00.499-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:00.499-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:00.661-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:00.661-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:01.003-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:01.003-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:01.164-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:01.164-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:01.505-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:01.505-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:01.667-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:01.667-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:02.006-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:02.006-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:02.006-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1701 checks in a row. 2016-11-02T14:20:02.171-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:02.171-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:02.508-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:02.508-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:02.674-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:02.674-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:03.013-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:03.013-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:03.180-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:03.180-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:03.518-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:03.518-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:03.686-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:03.686-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:04.020-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:04.020-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:04.187-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:04.187-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:04.523-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:04.523-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:04.523-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1711 checks in a row. 2016-11-02T14:20:04.693-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:04.693-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:05.026-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:05.026-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:05.194-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:05.194-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:05.530-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:05.530-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:05.530-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:20:05.695-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:05.695-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:06.196-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:06.196-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:06.702-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:06.702-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:07.207-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:07.207-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:07.712-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:07.713-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:07.713-0400 W SHARDING [Uptime reporter] failed to refresh mongos settings :: caused by :: FailedToSatisfyReadPreference: Failed to refresh the balancer settings due to FailedToSatisfyReadPreference: could not find host matching read preference { mode: "nearest" } for set csrs 2016-11-02T14:20:15.410-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:15.410-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:20:15.410-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] All nodes for set csrs are down. This has happened for 1721 checks in a row. 2016-11-02T14:20:15.914-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:15.914-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:16.416-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:16.416-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:16.587-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:16.587-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:16.919-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:16.919-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:17.089-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.089-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:17.421-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.421-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:17.593-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.593-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:17.719-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.719-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:17.922-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.922-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:18.094-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.094-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:18.094-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1731 checks in a row. 2016-11-02T14:20:18.222-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.222-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:18.423-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.423-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:18.600-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.600-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:18.723-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.723-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:18.928-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.928-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:19.106-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.106-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:19.225-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.225-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:19.431-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.431-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:19.610-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.610-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:19.726-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.726-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:19.726-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1741 checks in a row. 2016-11-02T14:20:19.936-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.936-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:20.115-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.116-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:20.229-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.229-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:20.440-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.440-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:20.617-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.617-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:20.731-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.732-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:20.943-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.944-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:21.118-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.118-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:21.234-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.234-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:21.445-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.445-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:21.445-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1751 checks in a row. 2016-11-02T14:20:21.624-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.624-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:21.738-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.738-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:21.949-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.949-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:22.127-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.128-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:22.242-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.243-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:22.455-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.455-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:22.629-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.629-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:22.743-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.743-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:22.960-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.960-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:23.130-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.130-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.130-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1761 checks in a row. 2016-11-02T14:20:23.248-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.248-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:23.462-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.463-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:23.631-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.631-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.754-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.754-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:23.964-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.964-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:24.132-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.132-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:24.255-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.255-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:24.464-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.465-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:24.635-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.635-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:24.756-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.756-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:24.756-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1771 checks in a row. 2016-11-02T14:20:24.970-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.970-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:25.140-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.140-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:25.261-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.262-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:25.474-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.474-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:25.644-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.644-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:25.765-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.765-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:25.976-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.976-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:26.148-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.148-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:26.266-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.266-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:26.478-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.478-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:26.478-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1781 checks in a row. 2016-11-02T14:20:26.652-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.652-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:26.768-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.768-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:26.979-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.979-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:27.153-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.153-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:27.269-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.269-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:27.480-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.480-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:27.657-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.657-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:27.771-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.771-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:27.986-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.986-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:28.162-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.162-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:28.162-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 1791 checks in a row. 2016-11-02T14:20:28.275-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.275-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:28.491-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.491-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:28.667-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.667-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:28.778-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.778-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:28.992-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.992-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:29.172-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.172-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:29.281-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.281-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:29.493-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.493-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:29.673-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.673-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:29.782-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.782-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:29.782-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1801 checks in a row. 2016-11-02T14:20:29.996-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.996-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:30.174-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.174-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:30.287-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.287-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:30.498-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.499-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:30.676-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.676-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:30.789-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.789-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:31.002-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.002-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:31.179-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.179-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:31.292-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.292-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:31.506-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.506-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:31.506-0400 I NETWORK [UserCacheInvalidator] All nodes for set csrs are down. This has happened for 1811 checks in a row. 2016-11-02T14:20:31.685-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.685-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:31.796-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.796-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:32.010-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.010-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:32.186-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.186-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:32.301-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.302-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:32.514-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.514-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:32.689-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.689-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:32.804-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.804-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:33.015-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.015-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:33.193-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.193-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:33.193-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 1821 checks in a row. 2016-11-02T14:20:33.306-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.306-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:33.517-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.517-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:33.697-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.697-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:33.810-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.811-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:34.021-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.021-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:34.202-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.202-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:34.315-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.315-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:34.522-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.522-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:34.706-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.706-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:34.816-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.816-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:34.816-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1831 checks in a row. 2016-11-02T14:20:35.024-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.025-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:35.212-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.212-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:35.318-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.318-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:35.527-0400 W NETWORK [UserCacheInvalidator] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.527-0400 W NETWORK [UserCacheInvalidator] No primary detected for set csrs 2016-11-02T14:20:35.527-0400 W ACCESS [UserCacheInvalidator] An error occurred while fetching current user cache generation to check if user cache needs invalidation: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primaryPreferred" } for set csrs 2016-11-02T14:20:35.718-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.718-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:35.823-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:35.823-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:36.223-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:36.223-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:36.324-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:36.324-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:36.725-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:36.725-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:36.725-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:20:36.725-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:20:36.826-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:36.826-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:36.826-0400 I NETWORK [Uptime reporter] All nodes for set csrs are down. This has happened for 1841 checks in a row. 2016-11-02T14:20:37.328-0400 W NETWORK [Uptime reporter] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:37.328-0400 W NETWORK [Uptime reporter] No primary detected for set csrs 2016-11-02T14:20:37.699-0400 I CONTROL [signalProcessingThread] got signal 2 (Interrupt: 2), will terminate after current cmd ends 2016-11-02T14:20:37.699-0400 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to liquid.local:27000 2016-11-02T14:20:37.699-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for liquid.local:27017:1478110245:2790712912708122139 :: caused by :: InterruptedAtShutdown: interrupted at shutdown 2016-11-02T14:20:37.699-0400 I CONTROL [signalProcessingThread] shutting down with code:0